QMP: Introduce specification file
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH 1/11 QMP: Introduce specification file
This file contains detailed QMP description and definitions.
--- monitor-protocol-spec.txt | 180 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 180 insertions(+), 0 deletions(-) create mode 100644 monitor-protocol-spec.txt
monitor-protocol-spec.txt new file +180 -0
@@ -0,0 +1,180 @@ + QEMU Monitor Protocol Specification - Version 0.1 + + Luiz Capitulino + <[email protected]> + +1. Introduction +=============== + +This document specifies the QEMU Monitor Protocol (QMP), a text-based protocol +which is available for applications to control QEMU at the machine-level. + +For a detailed list of supported commands, please, refer to file +monitor-protocol-commands.txt. + +2. Control Mode +=============== + +In order to enable QMP support, QEMU must start its Monitor subsystem in +"control mode" rather than the default "user mode". + +Currently, this is done by passing the following command-line option to +QEMU: + +-monitor control,<device> + +Please, refer to QEMU manpage or user-manual for more details. + +3. Protocol Specification +========================= + +This section details the protocol format. For the purpose of this document +"client" is any application that is communication with QEMU in control mode, +and "server" is QEMU itself. + +3.1 General definition +---------------------- + + o Only ASCII is permitted + o Case-insensitive + o All lines are terminated with CRLF + +3.2 Client Issued Commands +-------------------------- + + o Arguments are separated by single space + o Arguments, responses and errors are defined separately by each command + +3.3 Server responses +-------------------- + + There are four types of responses the server may send to the client: +greeting, command completion, command data and asynchronous messages. + +3.3.1 Server Greeting +--------------------- + +Sent when a new connection is opened. + +Format: + OK QEMU <version> QMP <version> +Example: + OK QEMU 0.10.50 QMP 0.1 + +3.3.2 Server Command Completion +------------------------------- + +Indicates that the execution of the last command has finished, there are +three possible responses, as outlined below. + + o Command completion was successful + + Format: + OK <command> completed + Example: + OK stop completed + + o Command completion failed + + Format: - ERR <reason> + Example: - ERR could not find network device 'foo' + + o Protocol error + + Format: - BAD <reason> + Example: - BAD unknown command + +3.3.3 Server Command Data +------------------------- + +Output issued by commands will always be prefixed by '='. If it is a +multiline response, then each line is prefixed by '='. + +For example, the output of 'info network' would be: + += VLAN 0 devices: += user.0: += ne2k_pci.0: model=ne2k_pci,macaddr=52:54:00:12:34:56 + +3.3.4 Asynchronous Messages +--------------------------- + +Messages that can be sent unilaterally by the server, at any time. They are +always prefixed by '*'. + +Currently there is only one type of asynchronous messages supported, +known as 'events'. + +Format: * EVENT <event> +Example: * EVENT reboot + +4. Examples +=========== + +This section provides some examples of the protocol in action, in all of +them 'C' stands for 'Client' and 'S' stands for 'Server'. + +4.1 Simple 'cont' Execution +--------------------------- + +Client started QEMU with the "-S" option, which makes QEMU do not start +the CPU at start up. Then Client issues the "cont" command. + +S: + OK QEMU 0.10.50 QMP 0.1 +C: cont +S: + OK cont completed + +4.2 Multiline Response +---------------------- + +Client queries the Server about the network. + +S: + OK QEMU 0.10.50 QMP 0.1 +C: info network +S: = VLAN 0 devices: +S: = user.0: +S: = ne2k_pci.0: model=ne2k_pci,macaddr=52:54:00:12:34:56 +S: + OK info network completed + +4.3 Events +---------- + +Client queries the Server about memory, but QEMU reboots. + +S: + OK QEMU 0.10.50 QMP 0.1 +C: info balloon +S: * EVENT reboot + +4.4 Protocol Error +------------------ + +Client issues an invalid command. + +S: + OK QEMU 0.10.50 QMP 0.1 +C: foo +S: - BAD unknown command + +4.5 Command Error +----------------- + +Client tries to change link status for a non existing device. + +S: + OK QEMU 0.10.50 QMP 0.1 +C: set_link foo down +S: - ERR could not find network device 'foo' + +5. Client Considerations +======================== + +This section contains some considerations for Client implementation. + + o If events are not supported, then lines starting with '*' should be + ignored and not considered errors + + o If events are supported, the Client must be prepared to deal with + events added in future protocol versions. The best way to do this is + to handle known events and ignore the others + +6. Protocol History +=================== + +v0.1 - 2009-06-19 +----------------- + + o Initial version
Signature · 1 line
1.6.3.GIT
Quoted 15 lines · Luiz Capitulino
This file contains detailed QMP description and definitions.
Signed-off-by Luiz Capitulino<[email protected]>--- monitor-protocol-spec.txt | 180 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 180 insertions(+), 0 deletions(-)create mode 100644 monitor-protocol-spec.txt
monitor-protocol-spec.txt new file +0
-0
monitor-protocol-spec.txt line 10
Quoted 10 lines
monitor-protocol-spec.txt new file +10
-0@@ -0,0 +1,180 @@ + QEMU Monitor Protocol Specification - Version 0.1 + + Luiz Capitulino +<[email protected]> + +1. Introduction +=============== + +This document specifies the QEMU Monitor Protocol (QMP), a text-based protocol +which is available for applications to control QEMU at the machine-level.
Without a doubt, this is a the most important file of this patchset. There's a huge difference between working with an implementation and working with a specification.
monitor-protocol-spec.txt line 13
Quoted 4 lines
monitor-protocol-spec.txt new file +3
-0+ +For a detailed list of supported commands, please, refer to file +monitor-protocol-commands.txt.
I don't see you update that file anywhere. In any case, my preference would be to have everything in one file.
monitor-protocol-spec.txt line 38
Quoted 5 lines
monitor-protocol-spec.txt new file +4
-0+3.1 General definition +---------------------- + + o Only ASCII is permitted
Since the some commands contain user-specified strings, UTF-8 is needed.
I think it's worthwhile to define a quoted string format, to be used both in commands and responses.
monitor-protocol-spec.txt line 60
Quoted 9 lines
monitor-protocol-spec.txt new file +8
-0+ +3.3.1 Server Greeting +--------------------- + +Sent when a new connection is opened. + +Format: + OK QEMU<version> QMP<version> +Example: + OK QEMU 0.10.50 QMP 0.1
Clients should never make decisions based on the qemu or qmp version. Rather, we should provide a facility to query the availability of features.
monitor-protocol-spec.txt line 77
Quoted 6 lines
monitor-protocol-spec.txt new file +5
-0+ o Command completion failed + + Format: - ERR<reason> + Example: - ERR could not find network device 'foo' +
Maybe add a numeric error code (to be defined by individual commands).
monitor-protocol-spec.txt line 142
Quoted 9 lines
monitor-protocol-spec.txt new file +8
-0+4.3 Events +---------- + +Client queries the Server about memory, but QEMU reboots. + +S: + OK QEMU 0.10.50 QMP 0.1 +C: info balloon +S: * EVENT reboot
The guest reboots (actually, resets), not qemu. And 'info balloon' will eventually print its response, no?
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 41 lines · Avi Kivity
Quoted 24 lines · Luiz Capitulino
This file contains detailed QMP description and definitions.
Signed-off-by Luiz Capitulino<[email protected]>--- monitor-protocol-spec.txt | 180 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 180 insertions(+), 0 deletions(-) create mode 100644 monitor-protocol-spec.txtmonitor-protocol-spec.txt new file +9
-0@@ -0,0 +1,180 @@ + QEMU Monitor Protocol Specification - Version 0.1 + + Luiz Capitulino +<[email protected]> + +1. Introduction +=============== + +This document specifies the QEMU Monitor Protocol (QMP), a text-basedQuoted 3 lines
protocol
+which is available for applications to control QEMU at the machine-level.Without a doubt, this is a the most important file of this patchset. There's a huge difference between working with an implementation and working with a specification.
monitor-protocol-spec.txt line 13
Quoted 4 lines
monitor-protocol-spec.txt new file +3
-0+ +For a detailed list of supported commands, please, refer to file +monitor-protocol-commands.txt.I don't see you update that file anywhere. In any case, my preference would be to have everything in one file.
Quoted 7 lines
Quoted 5 lines
+3.1 General definition
+----------------------
+
+ o Only ASCII is permittedSince the some commands contain user-specified strings, UTF-8 is needed.
This would imply that QEMU does charset conversion of all data going in or out of the monitor, to cope with the case of being run in a non-UTF8 locale. This is probably worth it though, because the only viable alternative would be to have the monitor always work in the locale the QEMU is launched under which pushes the problem off to every client.
Quoted 2 lines
I think it's worthwhile to define a quoted string format, to be used both in commands and responses.
Definitely need a quoted string format, space delimited is not sufficient to cope with filenames with spaces in them
Quoted 13 lines
monitor-protocol-spec.txt line 60
Quoted 10 lines
monitor-protocol-spec.txt new file +8
-0+ +3.3.1 Server Greeting +--------------------- + +Sent when a new connection is opened. + +Format: + OK QEMU<version> QMP<version> +Example: + OK QEMU 0.10.50 QMP 0.1Clients should never make decisions based on the qemu or qmp version. Rather, we should provide a facility to query the availability of features.
True, but having QEMU print full version info when connecting is still a useful thing for clients to log for developer debugging.
Quoted 8 lines
monitor-protocol-spec.txt line 77
Quoted 6 lines
+ o Command completion failed
monitor-protocol-spec.txt new file +4
-0+ + Format: - ERR<reason> + Example: - ERR could not find network device 'foo' +Maybe add a numeric error code (to be defined by individual commands).
I think that would be particularly useful to allow clients to distinguish the error from a command which does not exist, vs a command that exists but failed. There's probably a handful of common errors that you want to detect from all commands with the same error handling logic.
Quoted 13 lines
monitor-protocol-spec.txt line 142
Quoted 10 lines
+4.3 Events
+----------monitor-protocol-spec.txt new file +6
-0+ +Client queries the Server about memory, but QEMU reboots. + +S: + OK QEMU 0.10.50 QMP 0.1 +C: info balloon +S: * EVENT rebootThe guest reboots (actually, resets), not qemu. And 'info balloon' will eventually print its response, no?
Might we need to have timestamp assoicated with each response, or will we define that responses are explicitly ordered wrt events,to reflect the order in which they're handled. eg When the 'info balloon' response arrives, we want to know whether the data it contains is reflecting state before or after the reboot.
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Quoted 8 lines · Daniel P. Berrange
Quoted 2 lines
Maybe add a numeric error code (to be defined by individual commands).
I think that would be particularly useful to allow clients to distinguish the error from a command which does not exist, vs a command that exists but failed. There's probably a handful of common errors that you want to detect from all commands with the same error handling logic.
There should be mechanisms to detect which commands are available. Not all commands will have fallbacks, so the user might want to query which commands (or features) exist before actually launching the guest.
Quoted 10 lines
Quoted 3 lines
The guest reboots (actually, resets), not qemu. And 'info balloon' will eventually print its response, no?
Might we need to have timestamp assoicated with each response, or will we define that responses are explicitly ordered wrt events,to reflect the order in which they're handled. eg When the 'info balloon' response arrives, we want to know whether the data it contains is reflecting state before or after the reboot.
Responses are implicitly ordered. But a timestamp for events might be good for other reasons (when the event happened vs. when management got around to processing it; might be good to correlate with system events).
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · On Tue, 23 Jun 2009 10:57:54 +0100 "Daniel P. Berrange" <[email protected]>…
Quoted 8 lines
monitor-protocol-spec.txt line 77
Quoted 6 lines
+ o Command completion failed
monitor-protocol-spec.txt new file +4
-0+ + Format: - ERR<reason> + Example: - ERR could not find network device 'foo' +Maybe add a numeric error code (to be defined by individual commands).
Quoted 4 lines
I think that would be particularly useful to allow clients to distinguish the error from a command which does not exist, vs a command that exists but failed. There's probably a handful of common errors that you want to detect from all commands with the same error handling logic.
I have followed IMAP's design on this, which specifies that commands
execution errors are "- ERR" and protocol errors are "- BAD".
So, commands that don't exist are protocol errors and applications
can distinguish them.
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
+4.3 Events
+----------(quoted diff) +2
-0+ +Client queries the Server about memory, but QEMU reboots.Quoted 8 lines
monitor-protocol-spec.txt line 142
Quoted 5 lines
monitor-protocol-spec.txt new file +4
-0+ +S: + OK QEMU 0.10.50 QMP 0.1 +C: info balloon +S: * EVENT rebootThe guest reboots (actually, resets), not qemu. And 'info balloon' will eventually print its response, no?
Quoted 5 lines
Might we need to have timestamp assoicated with each response, or will we define that responses are explicitly ordered wrt events,to reflect the order in which they're handled. eg When the 'info balloon' response arrives, we want to know whether the data it contains is reflecting state before or after the reboot.
They will be ordered, but I liked the timestamp idea. Do you think it's
going to useful only for events, as suggested by Avi?
Quoted 4 lines · Luiz Capitulino
They will be ordered, but I liked the timestamp idea. Do you think it's
going to useful only for events, as suggested by Avi?
Might be useful for synchronous responses as well.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 22 lines · On Tue, 23 Jun 2009 11:55:56 +0300 Avi Kivity <[email protected]> wrote:
Quoted 21 lines · Luiz Capitulino
This file contains detailed QMP description and definitions.
Signed-off-by Luiz Capitulino<[email protected]>--- monitor-protocol-spec.txt | 180 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 180 insertions(+), 0 deletions(-)create mode 100644 monitor-protocol-spec.txt
monitor-protocol-spec.txt new file +7
-0@@ -0,0 +1,180 @@ + QEMU Monitor Protocol Specification - Version 0.1 + + Luiz Capitulino +<[email protected]> + +1. Introduction +===============
Quoted 8 lines
monitor-protocol-spec.txt line 10
Quoted 4 lines
monitor-protocol-spec.txt new file +3
-0+ +This document specifies the QEMU Monitor Protocol (QMP), a text-based protocol +which is available for applications to control QEMU at the machine-level.Without a doubt, this is a the most important file of this patchset. There's a huge difference between working with an implementation and working with a specification.
Exactly.
Quoted 7 lines
monitor-protocol-spec.txt line 13
Quoted 4 lines
monitor-protocol-spec.txt new file +3
-0+ +For a detailed list of supported commands, please, refer to file +monitor-protocol-commands.txt.I don't see you update that file anywhere. In any case, my preference would be to have everything in one file.
Sorry, I forgot to mention that I wouldn't include this file
in the patchset.
But its format will be like POP3 or IMAP specs, something like:
"""
6.2 cont
Arguments: none.
Responses: + OK cont completed
<description>
<example>
"""
We can have everything in the same file then.
Quoted 10 lines
monitor-protocol-spec.txt line 38
Quoted 5 lines
monitor-protocol-spec.txt new file +4
-0+3.1 General definition +---------------------- + + o Only ASCII is permittedSince the some commands contain user-specified strings, UTF-8 is needed.
I think it's worthwhile to define a quoted string format, to be used both in commands and responses.
Ok, as Daniel also said we will need this.
The only problem is that I was trying to avoid changing commands'
output too much, because this can turn into a maintenance nightmare.
Note that we will have to maintain two protocols: the current existing
(user mode) and this new control mode.
Quoted 4 lines
Quoted 4 lines
monitor-protocol-spec.txt new file +4
-0+ +3.3.1 Server Greeting +--------------------- +
Quoted 8 lines
monitor-protocol-spec.txt line 60
Quoted 5 lines
monitor-protocol-spec.txt new file +4
-0+Sent when a new connection is opened. + +Format: + OK QEMU<version> QMP<version> +Example: + OK QEMU 0.10.50 QMP 0.1Clients should never make decisions based on the qemu or qmp version. Rather, we should provide a facility to query the availability of features.
Right. What about changing 'help' output to only list command names?
Quoted 8 lines
monitor-protocol-spec.txt line 77
Quoted 6 lines
monitor-protocol-spec.txt new file +5
-0+ o Command completion failed + + Format: - ERR<reason> + Example: - ERR could not find network device 'foo' +Maybe add a numeric error code (to be defined by individual commands).
Well, numeric error codes seem redundant in text protocols. At least,
POP3 and IMAP don't use them.
This may seem contradictory, but although this a low-level protocol,
I feel that text protocols tend to be very descriptive.
Quoted 4 lines
Quoted 4 lines
monitor-protocol-spec.txt new file +4
-0+4.3 Events +---------- + +Client queries the Server about memory, but QEMU reboots.
Quoted 8 lines
monitor-protocol-spec.txt line 142
Quoted 5 lines
monitor-protocol-spec.txt new file +4
-0+ +S: + OK QEMU 0.10.50 QMP 0.1 +C: info balloon +S: * EVENT rebootThe guest reboots (actually, resets), not qemu. And 'info balloon' will eventually print its response, no?
Yes, it will. I will fix this.
Quoted 106 lines · Luiz Capitulino
Quoted 22 lines · On Tue, 23 Jun 2009 11:55:56 +0300 Avi Kivity <[email protected]> wrote:
Quoted 21 lines · Luiz Capitulino
This file contains detailed QMP description and definitions.
Signed-off-by Luiz Capitulino<[email protected]>--- monitor-protocol-spec.txt | 180 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 180 insertions(+), 0 deletions(-)create mode 100644 monitor-protocol-spec.txt
monitor-protocol-spec.txt new file +7
-0@@ -0,0 +1,180 @@ + QEMU Monitor Protocol Specification - Version 0.1 + + Luiz Capitulino +<[email protected]> + +1. Introduction +===============Quoted 8 lines
monitor-protocol-spec.txt line 10
Quoted 4 lines
monitor-protocol-spec.txt new file +3
-0+ +This document specifies the QEMU Monitor Protocol (QMP), a text-based protocol +which is available for applications to control QEMU at the machine-level.Without a doubt, this is a the most important file of this patchset. There's a huge difference between working with an implementation and working with a specification.
Exactly.
Quoted 7 lines
monitor-protocol-spec.txt line 13
Quoted 4 lines
monitor-protocol-spec.txt new file +3
-0+ +For a detailed list of supported commands, please, refer to file +monitor-protocol-commands.txt.I don't see you update that file anywhere. In any case, my preference would be to have everything in one file.
Sorry, I forgot to mention that I wouldn't include this file
in the patchset.But its format will be like POP3 or IMAP specs, something like:
"""
6.2 contArguments: none.
Responses: + OK cont completed
<description>
<example>
"""We can have everything in the same file then.
Quoted 10 lines
monitor-protocol-spec.txt line 38
Quoted 5 lines
monitor-protocol-spec.txt new file +4
-0+3.1 General definition +---------------------- + + o Only ASCII is permittedSince the some commands contain user-specified strings, UTF-8 is needed.
I think it's worthwhile to define a quoted string format, to be used both in commands and responses.
Ok, as Daniel also said we will need this.
The only problem is that I was trying to avoid changing commands'
output too much, because this can turn into a maintenance nightmare.Note that we will have to maintain two protocols: the current existing
(user mode) and this new control mode.Quoted 4 lines
Quoted 4 lines
monitor-protocol-spec.txt new file +4
-0+ +3.3.1 Server Greeting +--------------------- +Quoted 8 lines
monitor-protocol-spec.txt line 60
Quoted 5 lines
monitor-protocol-spec.txt new file +4
-0+Sent when a new connection is opened. + +Format: + OK QEMU<version> QMP<version> +Example: + OK QEMU 0.10.50 QMP 0.1Clients should never make decisions based on the qemu or qmp version. Rather, we should provide a facility to query the availability of features.
Right. What about changing 'help' output to only list command names?
Quoted 8 lines
monitor-protocol-spec.txt line 77
Quoted 6 lines
monitor-protocol-spec.txt new file +5
-0+ o Command completion failed + + Format: - ERR<reason> + Example: - ERR could not find network device 'foo' +Maybe add a numeric error code (to be defined by individual commands).
Quoted 2 lines
Well, numeric error codes seem redundant in text protocols. At least,
POP3 and IMAP don't use them.
I counter with FTP and HTTP, which do :-P
eg http://www.faqs.org/rfcs/rfc959.html
200 Command okay.
500 Syntax error, command unrecognized.
This may include errors such as command line too long.
501 Syntax error in parameters or arguments. 202 Command not implemented, superfluous at this site. 502 Command not implemented.
503 Bad sequence of commands.
504 Command not implemented for that parameter.
Regards,
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Quoted 114 lines · On Tue, 23 Jun 2009 14:10:34 +0100 "Daniel P. Berrange" <[email protected]>…
Quoted 106 lines · Luiz Capitulino
Quoted 22 lines · On Tue, 23 Jun 2009 11:55:56 +0300 Avi Kivity <[email protected]> wrote:
Quoted 21 lines · Luiz Capitulino
This file contains detailed QMP description and definitions.
Signed-off-by Luiz Capitulino<[email protected]>--- monitor-protocol-spec.txt | 180 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 180 insertions(+), 0 deletions(-)create mode 100644 monitor-protocol-spec.txt
monitor-protocol-spec.txt new file +7
-0@@ -0,0 +1,180 @@ + QEMU Monitor Protocol Specification - Version 0.1 + + Luiz Capitulino +<[email protected]> + +1. Introduction +===============Quoted 8 lines
monitor-protocol-spec.txt line 10
Quoted 4 lines
monitor-protocol-spec.txt new file +3
-0+ +This document specifies the QEMU Monitor Protocol (QMP), a text-based protocol +which is available for applications to control QEMU at the machine-level.Without a doubt, this is a the most important file of this patchset. There's a huge difference between working with an implementation and working with a specification.
Exactly.
Quoted 7 lines
monitor-protocol-spec.txt line 13
Quoted 4 lines
monitor-protocol-spec.txt new file +3
-0+ +For a detailed list of supported commands, please, refer to file +monitor-protocol-commands.txt.I don't see you update that file anywhere. In any case, my preference would be to have everything in one file.
Sorry, I forgot to mention that I wouldn't include this file
in the patchset.But its format will be like POP3 or IMAP specs, something like:
"""
6.2 contArguments: none.
Responses: + OK cont completed
<description>
<example>
"""We can have everything in the same file then.
Quoted 10 lines
monitor-protocol-spec.txt line 38
Quoted 5 lines
monitor-protocol-spec.txt new file +4
-0+3.1 General definition +---------------------- + + o Only ASCII is permittedSince the some commands contain user-specified strings, UTF-8 is needed.
I think it's worthwhile to define a quoted string format, to be used both in commands and responses.
Ok, as Daniel also said we will need this.
The only problem is that I was trying to avoid changing commands'
output too much, because this can turn into a maintenance nightmare.Note that we will have to maintain two protocols: the current existing
(user mode) and this new control mode.Quoted 4 lines
Quoted 4 lines
monitor-protocol-spec.txt new file +4
-0+ +3.3.1 Server Greeting +--------------------- +Quoted 8 lines
monitor-protocol-spec.txt line 60
Quoted 5 lines
monitor-protocol-spec.txt new file +4
-0+Sent when a new connection is opened. + +Format: + OK QEMU<version> QMP<version> +Example: + OK QEMU 0.10.50 QMP 0.1Clients should never make decisions based on the qemu or qmp version. Rather, we should provide a facility to query the availability of features.
Right. What about changing 'help' output to only list command names?
Quoted 8 lines
monitor-protocol-spec.txt line 77
Quoted 6 lines
monitor-protocol-spec.txt new file +5
-0+ o Command completion failed + + Format: - ERR<reason> + Example: - ERR could not find network device 'foo' +Maybe add a numeric error code (to be defined by individual commands).
Quoted 2 lines
Well, numeric error codes seem redundant in text protocols. At least,
POP3 and IMAP don't use them.I counter with FTP and HTTP, which do :-P
Quoted 8 lines
200 Command okay.
500 Syntax error, command unrecognized.
This may include errors such as command line too long.
501 Syntax error in parameters or arguments. 202 Command not implemented, superfluous at this site. 502 Command not implemented.
503 Bad sequence of commands.
504 Command not implemented for that parameter.
There goes my argument. :)
Anthony seems to prefer this way too, although I still think
this is a bit redundant, I don't have a strong opinion here.
So, we can do this way then.
Quoted 7 lines · Luiz Capitulino
Quoted 3 lines
I don't see you update that file anywhere. In any case, my preference would be to have everything in one file.
Sorry, I forgot to mention that I wouldn't include this file
in the patchset.
You mean now, or forever?
Quoted 8 lines
Ok, as Daniel also said we will need this.
The only problem is that I was trying to avoid changing commands'
output too much, because this can turn into a maintenance nightmare.Note that we will have to maintain two protocols: the current existing
(user mode) and this new control mode.
I don't see a way out. The current protocol is not machine consumable, but we can't change it. We'll just have to live with it.
Quoted 6 lines
Quoted 3 lines
Clients should never make decisions based on the qemu or qmp version. Rather, we should provide a facility to query the availability of features.
Right. What about changing 'help' output to only list command names?
Also command subfeatures, when new subfeatures are added.
Quoted 10 lines
Quoted 3 lines
Maybe add a numeric error code (to be defined by individual commands).
Well, numeric error codes seem redundant in text protocols. At least,
POP3 and IMAP don't use them.This may seem contradictory, but although this a low-level protocol,
I feel that text protocols tend to be very descriptive.
Well, the strings are not single words, so the key for the application is "all words until the end of the line". Not too machine friendly.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · On Tue, 23 Jun 2009 16:27:16 +0300 Avi Kivity <[email protected]> wrote:
Quoted 7 lines · Luiz Capitulino
Quoted 3 lines
I don't see you update that file anywhere. In any case, my preference would be to have everything in one file.
Sorry, I forgot to mention that I wouldn't include this file
in the patchset.You mean now, or forever?
Only for now, as you noted in another email command porting/additions
should be NACKed if not properly documented.
Quoted 11 lines
Quoted 8 lines
Ok, as Daniel also said we will need this.
The only problem is that I was trying to avoid changing commands'
output too much, because this can turn into a maintenance nightmare.Note that we will have to maintain two protocols: the current existing
(user mode) and this new control mode.I don't see a way out. The current protocol is not machine consumable, but we can't change it. We'll just have to live with it.
Yes.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Clients should never make decisions based on the qemu or qmp version. Rather, we should provide a facility to query the availability of features.
Right. What about changing 'help' output to only list command names?
Also command subfeatures, when new subfeatures are added.
Ok, you suggested a format I'm probably going to use.
Quoted 13 lines
Quoted 9 lines
Quoted 2 lines
Maybe add a numeric error code (to be defined by individual commands).
Well, numeric error codes seem redundant in text protocols. At least,
POP3 and IMAP don't use them.This may seem contradictory, but although this a low-level protocol,
I feel that text protocols tend to be very descriptive.Well, the strings are not single words, so the key for the application is "all words until the end of the line". Not too machine friendly.
Hi Luiz,
The specification looks pretty good.
Quoted 11 lines
monitor-protocol-spec.txt line 60
Quoted 9 lines
monitor-protocol-spec.txt new file +8
-0+ +3.3.1 Server Greeting +--------------------- + +Sent when a new connection is opened. + +Format: + OK QEMU<version> QMP<version> +Example: + OK QEMU 0.10.50 QMP 0.1Clients should never make decisions based on the qemu or qmp version. Rather, we should provide a facility to query the availability of features.
I agree, but I'd suggest leaving the QMP version in there for insurance purposes in case we really screw up and need to bump the version. In fact, having the client also negotiate the QMP version isn't a bad idea.
Quoted 8 lines
monitor-protocol-spec.txt line 77
Quoted 6 lines
+ o Command completion failed
monitor-protocol-spec.txt new file +4
-0+ + Format: - ERR<reason> + Example: - ERR could not find network device 'foo' +Maybe add a numeric error code (to be defined by individual commands).
I think copying HTTP makes sense here.
How would asynchronous commands work? Could you give an example of doing live migration through QMP?
Regards,
Anthony Liguori
Quoted 16 lines · Anthony Liguori
Hi Luiz,
The specification looks pretty good.
Quoted 10 lines
monitor-protocol-spec.txt line 60
Quoted 8 lines
monitor-protocol-spec.txt new file +8
-0+ +3.3.1 Server Greeting +--------------------- + +Sent when a new connection is opened. + +Format: + OK QEMU<version> QMP<version> +Example: + OK QEMU 0.10.50 QMP 0.1Clients should never make decisions based on the qemu or qmp version. Rather, we should provide a facility to query the availability of features.
Quoted 1 line
I agree, but I'd suggest leaving the QMP version in there for insurance purposes in case we really screw up and need to bump the version. In fact, having the client also negotiate the QMP version isn't a bad idea.
Agreed.
Quoted 1 line
How would asynchronous commands work?
IMO, there aren't any. All commands are synchronous (but may cause events to be generated later).
Quoted 2 lines
Could you give an example of doing live migration through QMP?
migrate tcp:blah:4444
+OK don't you love us anymore?
* EVENT stopped (migration)
* EVENT migration completed
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 24 lines · Avi Kivity
Quoted 18 lines · Anthony Liguori
Hi Luiz,
The specification looks pretty good.
Quoted 4 lines
Quoted 4 lines
monitor-protocol-spec.txt new file +4
-0+ +3.3.1 Server Greeting +--------------------- +Quoted 8 lines
monitor-protocol-spec.txt line 60
Quoted 4 lines
monitor-protocol-spec.txt new file +4
-0+Sent when a new connection is opened. + +Format: + OK QEMU<version> QMP<version> +Example: + OK QEMU 0.10.50 QMP 0.1Clients should never make decisions based on the qemu or qmp version. Rather, we should provide a facility to query the availability of features.
Quoted 4 lines
I agree, but I'd suggest leaving the QMP version in there for insurance purposes in case we really screw up and need to bump the version. In fact, having the client also negotiate the QMP version isn't a bad idea.
Quoted 6 lines
Agreed.
Quoted 1 line
How would asynchronous commands work?
IMO, there aren't any. All commands are synchronous (but may cause events to be generated later).
And if there is any, we can simply convert them to synchronous commands that cause events to be generated.
I think we could make this clear on the specification, specifying that all commands are synchronous.
Signature · 1 line
Eduardo
Quoted 17 lines · Avi Kivity
Quoted 16 lines · Anthony Liguori
Hi Luiz,
The specification looks pretty good.
Quoted 10 lines
monitor-protocol-spec.txt line 60
Quoted 8 lines
monitor-protocol-spec.txt new file +8
-0+ +3.3.1 Server Greeting +--------------------- + +Sent when a new connection is opened. + +Format: + OK QEMU<version> QMP<version> +Example: + OK QEMU 0.10.50 QMP 0.1Clients should never make decisions based on the qemu or qmp version. Rather, we should provide a facility to query the availability of features.
Quoted 7 lines
Quoted 1 line
I agree, but I'd suggest leaving the QMP version in there for insurance purposes in case we really screw up and need to bump the version. In fact, having the client also negotiate the QMP version isn't a bad idea.
Agreed.
Quoted 1 line
How would asynchronous commands work?
IMO, there aren't any. All commands are synchronous (but may cause events to be generated later).
At least today, there are quite a few commands that are synchronous and block. They would all have to be converted to events. For instance:
commit
change
screendump
savevm
loadvm
delvm
stop
cont
memsave
pmemsave
migrate
balloon
Regards,
Anthony Liguori
Quoted 5 lines · Anthony Liguori
Quoted 1 line
IMO, there aren't any. All commands are synchronous (but may cause events to be generated later).
At least today, there are quite a few commands that are synchronous and block. They would all have to be converted to events. For instance:
That's what I meant, though I phrased it poorly.
Quoted 1 line
commit
That's a biggie to make truly async.
Quoted 2 lines
change
screendump
I think we can keep those synchronous.
Quoted 3 lines
savevm
loadvm
delvm
Like commit, very difficult to make async.
Quoted 2 lines
stop
cont
These certainly seem very synchronous to me.
Quoted 2 lines
memsave
pmemsave
Can keep these synchronous IMO. You certainly don't want the guest to progress during memsave.
Quoted 2 lines
migrate
balloon
Isn't balloon synchronous? It sets the target and exits immediately. We don't want command completion to depend on the guest.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 4 lines · Avi Kivity
Quoted 2 lines
stop
contThese certainly seem very synchronous to me.
Not with KVM. You have to wait for the VCPU threads to get notified and stop. Maybe it's not a big deal to block during this.
Quoted 4 lines
Quoted 2 lines
migrate
balloonIsn't balloon synchronous? It sets the target and exits immediately. We don't want command completion to depend on the guest.
I can see a use-case for having balloon only return when the target has been reached by the guest.
I think I can buy the event stuff. As long as we version QMP protocol, we can potentially fix this down the road.
Signature · 3 lines
Regards, Anthony Liguori
Quoted 7 lines · Anthony Liguori
Quoted 4 lines · Avi Kivity
Quoted 2 lines
stop
contThese certainly seem very synchronous to me.
Not with KVM. You have to wait for the VCPU threads to get notified and stop. Maybe it's not a big deal to block during this.
If a command returns immediately, we shouldn't try to make it asynchronous.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
migrate
balloonIsn't balloon synchronous? It sets the target and exits immediately. We don't want command completion to depend on the guest.
I can see a use-case for having balloon only return when the target has been reached by the guest.
There's 'info balloon' for that, and maybe request a notification. Waiting for the guest is asking for trouble. What if you want to set the balloon to a different target? you have two different balloon commands running?
Quoted 3 lines
I think I can buy the event stuff. As long as we version QMP protocol, we can potentially fix this down the road.
Alternatively we can introduce a tag, and so make all commands asynchronous:
@tag1 commit
@tag1 = committed
@tag1 + OK have fun
Or go with the jsonrpc thing that has all this built in.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 1 line · On Tue, 23 Jun 2009 08:12:33 -0500 Anthony Liguori <[email protected]>…
The specification looks pretty good.
Thanks, used previous discussions as my main reference.
Quoted 14 lines
Quoted 5 lines
Quoted 5 lines
monitor-protocol-spec.txt new file +5
-0+ +3.3.1 Server Greeting +--------------------- + +Sent when a new connection is opened.Quoted 8 lines
monitor-protocol-spec.txt line 60
Quoted 4 lines
monitor-protocol-spec.txt new file +3
-0+ +Format: + OK QEMU<version> QMP<version> +Example: + OK QEMU 0.10.50 QMP 0.1Clients should never make decisions based on the qemu or qmp version. Rather, we should provide a facility to query the availability of features.
Quoted 3 lines
I agree, but I'd suggest leaving the QMP version in there for insurance purposes in case we really screw up and need to bump the version. In fact, having the client also negotiate the QMP version isn't a bad idea.
Will keep.
Quoted 10 lines
Quoted 8 lines
monitor-protocol-spec.txt line 77
Quoted 6 lines
+ o Command completion failed
monitor-protocol-spec.txt new file +4
-0+ + Format: - ERR<reason> + Example: - ERR could not find network device 'foo' +Maybe add a numeric error code (to be defined by individual commands).
I think copying HTTP makes sense here.
Ok, as I said in others emails I was a bit agains't it but have
changed my mind after three being in favor. :)
Quoted 2 lines
How would asynchronous commands work? Could you give an example of doing live migration through QMP?
Avi has already explained this part. To be honest, I didn't have this
clear in my mind, but Avi's suggestion is what makes sense to me.
Quoted 14 lines · Luiz Capitulino
This file contains detailed QMP description and definitions.
Signed-off-by Luiz Capitulino <[email protected]>--- monitor-protocol-spec.txt | 180 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 180 insertions(+), 0 deletions(-) create mode 100644 monitor-protocol-spec.txtmonitor-protocol-spec.txt new file +0
-0
monitor-protocol-spec.txt line 1
Quoted 1 line
monitor-protocol-spec.txt new file +1
-0@@ -0,0 +1,180 @@ + QEMU Monitor Protocol Specification - Version 0.1
I think this is extremelly needed however inventing yet another RPC protocol make no sense in 2009, specially with all the limitations (no UTF-8, no quoted string ...).
You should really look at what's already out there, so you don't have to concentrate on the marshalled protocol, but just the format.
At this point my personal suggestion would be jsonrpc:
- it's easy to parse (even in C)
- it's text with markup
- easy to generate
- support utf-8
- quoted string
- it's not ugly to read (compare to xml)
- can be embedded directly (no more library dependancy for qemu) in the code
- lots of language have json library making the interaction with the monitor easy from the other side
However i'm sure other RPC "format" exists that suit all of these too, but again please don't do something from scratch ..
Signature · 1 line
Vincent
Quoted 28 lines · Vincent Hanquez
Quoted 15 lines · Luiz Capitulino
This file contains detailed QMP description and definitions.
Signed-off-by Luiz Capitulino<[email protected]>--- monitor-protocol-spec.txt | 180 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 180 insertions(+), 0 deletions(-)create mode 100644 monitor-protocol-spec.txt
monitor-protocol-spec.txt new file +0
-0monitor-protocol-spec.txt line 1
Quoted 1 line
monitor-protocol-spec.txt new file +1
-0@@ -0,0 +1,180 @@ + QEMU Monitor Protocol Specification - Version 0.1I think this is extremelly needed however inventing yet another RPC protocol make no sense in 2009, specially with all the limitations (no UTF-8, no quoted string ...).
You should really look at what's already out there, so you don't have to concentrate on the marshalled protocol, but just the format.
At this point my personal suggestion would be jsonrpc:
Quoted 10 lines
- it's easy to parse (even in C)
- it's text with markup
- easy to generate
- support utf-8
- quoted string
- it's not ugly to read (compare to xml)
- can be embedded directly (no more library dependancy for qemu) in the code
- lots of language have json library making the interaction with the monitor easy from the other side
- supports passing structure
- supports notifications
- supports asynchronous commands
seems quite nice.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 17 lines · Avi Kivity
Quoted 15 lines · Vincent Hanquez
At this point my personal suggestion would be jsonrpc:
- it's easy to parse (even in C)
- it's text with markup
- easy to generate
- support utf-8
- quoted string
- it's not ugly to read (compare to xml)
- can be embedded directly (no more library dependancy for qemu) in the code
- lots of language have json library making the interaction with the monitor
easy from the other side
Quoted 5 lines
- supports passing structure
- supports notifications
- supports asynchronous commandsseems quite nice.
I'd rather not do json.
As they stand, I think the current patch set is very close to being mergable. I'd rather not go off on a new tangent.
Regards,
Anthony Liguori
Signature · 3 lines
Regards, Anthony Liguori
Quoted 17 lines · Anthony Liguori
Quoted 16 lines · Avi Kivity
Quoted 14 lines · Vincent Hanquez
At this point my personal suggestion would be jsonrpc:
- it's easy to parse (even in C)
- it's text with markup
- easy to generate
- support utf-8
- quoted string
- it's not ugly to read (compare to xml)
- can be embedded directly (no more library dependancy for qemu) in the code
- lots of language have json library making the interaction with the monitor
easy from the other side
Quoted 8 lines
Quoted 5 lines
- supports passing structure
- supports notifications
- supports asynchronous commandsseems quite nice.
I'd rather not do json.
Why not?
Quoted 1 line
As they stand, I think the current patch set is very close to being mergable. I'd rather not go off on a new tangent.
It fulfils all are requirements and more. There's a C implementation available. We can concentrate on the commands and ignore the plumbing.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 4 lines · Avi Kivity
Quoted 2 lines
I'd rather not do json.
Why not?
Because it's too much complexity at this level in the stack.
Quoted 3 lines
Quoted 1 line
As they stand, I think the current patch set is very close to being mergable. I'd rather not go off on a new tangent.
It fulfils all are requirements and more. There's a C implementation available. We can concentrate on the commands and ignore the plumbing.
The same is true for XML-RPC, CIM, dbus, etc. RPC grow like weeds. In 5 years, it will be shocking that qemu uses jsonrpc instead of the next super-neat RPC protocol.
If we did a real RPC mechanism, I'd want to stick with something tried-and-true like XDR. But an RPC mechanism is not the monitor. If we were going to do that, there's no point in morphing the monitor into that.
What we're doing here is adding a computer-mode to the existing monitor. This will be easy for current apps to consume and for us to transition the code base to. It's a safe and incremental improvement.
I don't want to invent Xen-API here and go through that pain in QEMU.
Signature · 3 lines
Regards, Anthony Liguori
Quoted 7 lines · Anthony Liguori
Quoted 4 lines · Avi Kivity
Quoted 2 lines
I'd rather not do json.
Why not?
Because it's too much complexity at this level in the stack.
I don't see that. Just another library.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
As they stand, I think the current patch set is very close to being mergable. I'd rather not go off on a new tangent.
It fulfils all are requirements and more. There's a C implementation available. We can concentrate on the commands and ignore the plumbing.
The same is true for XML-RPC, CIM, dbus, etc. RPC grow like weeds. In 5 years, it will be shocking that qemu uses jsonrpc instead of the next super-neat RPC protocol.
The only one that actually competes here is xml-rpc, which is ridiculously verbose.
Quoted 1 line
If we did a real RPC mechanism, I'd want to stick with something tried-and-true like XDR.
Having some experience with xdr, I'd call it tried-and-wrong. It's simply horrible.
Quoted 4 lines
But an RPC mechanism is not the monitor. If we were going to do
that, there's no point in morphing the monitor into that.What we're doing here is adding a computer-mode to the existing monitor. This will be easy for current apps to consume and for us to transition the code base to. It's a safe and incremental improvement.
An RPC makes it a lot easier for clients to parse, there's no need for a regular expression per command, or to bother with line continuations etc. You just focus on the data.
We have an opportunity to make a clean break, let's make use of it.
Quoted 1 line
I don't want to invent Xen-API here and go through that pain in QEMU.
I don't see the parallel.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 4 lines · Avi Kivity
An RPC makes it a lot easier for clients to parse, there's no need for a regular expression per command, or to bother with line continuations etc. You just focus on the data.
We have an opportunity to make a clean break, let's make use of it.
If we structure things internally correctly, by having a flow that basically looks like:
int <monitor_command>(Monitor *mon, <args>)
{
// process args
if (success) {
monitor_print_data(mon, <highly structured output>);
} else {
record_extended_error(<message>);
return error_code; }
}
Then it that works well for both the human monitor and the computer monitor. It's an RPC though.
Should we support other types of RPCs? I don't know. I'd think that you could create a computer monitor -> jsonrpc bridge easily enough.
If we implement this correctly, then an RPC interface could just be another option to see the computer monitor mode implemented as a "module".
Quoted 3 lines
Quoted 1 line
I don't want to invent Xen-API here and go through that pain in QEMU.
I don't see the parallel.
Xen-API was the "clean break" for Xend.
Signature · 3 lines
Regards, Anthony Liguori
Quoted 16 lines · Anthony Liguori
Quoted 4 lines · Avi Kivity
An RPC makes it a lot easier for clients to parse, there's no need for a regular expression per command, or to bother with line continuations etc. You just focus on the data.
We have an opportunity to make a clean break, let's make use of it.
If we structure things internally correctly, by having a flow that basically looks like:
int <monitor_command>(Monitor *mon, <args>)
{
// process argsif (success) { monitor_print_data(mon, <highly structured output>); } else { record_extended_error(<message>); return error_code;
Quoted 4 lines
}
}Then it that works well for both the human monitor and the computer monitor. It's an RPC though.
It looks brittle to me. What if you want to add an array? Some structure that contains a nested structure?
Quoted 1 line
Should we support other types of RPCs? I don't know. I'd think that you could create a computer monitor -> jsonrpc bridge easily enough.
No doubt, but I look at jsonrpc as a way of reducing the effort in qemu. Not so much interested in it for its own sake.
Quoted 1 line
If we implement this correctly, then an RPC interface could just be another option to see the computer monitor mode implemented as a "module".
This is deep into overkill territory IMO. We should pick some format that is good enough and has implementations for a variety of languages. Luiz' RFC was well done, but the subject is complex enough to be difficult to get right. And as far as I could tell, all the issues are addressed by jsonrpc. The only downside I can see is that jsonrpc is some kind of ajax-kiddie web-2.0 script dom thingie while we are a serious UNIX style daemon that complies with RFC 123456.
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
I don't want to invent Xen-API here and go through that pain in QEMU.
I don't see the parallel.
Xen-API was the "clean break" for Xend.
Anything can be screwed up. Normally I'm very much against rewrites and totally for incremental progress, but when it comes to protocols, you have to get them right first shot.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 1 line · Avi Kivity
It looks brittle to me. What if you want to add an array? Some structure that contains a nested structure?
Can you give me concrete examples?
The typical way this would be handled in an RPC library in C is to introduce complex types. For instance, you could have a MonitorList and a corresponding MON_LIST type. FWIW, is there a jsonrpc library for C that's actually maintained?
I can't find one that even has a functioning website.
Quoted 3 lines
Quoted 1 line
If we implement this correctly, then an RPC interface could just be another option to see the computer monitor mode implemented as a "module".
This is deep into overkill territory IMO. We should pick some format that is good enough and has implementations for a variety of languages. Luiz' RFC was well done, but the subject is complex enough to be difficult to get right. And as far as I could tell, all the issues are addressed by jsonrpc. The only downside I can see is that jsonrpc is some kind of ajax-kiddie web-2.0 script dom thingie while we are a serious UNIX style daemon that complies with RFC 123456.
There are many reasons why JSON is a bad representation. For starters, there is no way to encode NULL values in a JSON string. This means you cannot send binary data unless you base64 encode it first. The bounds of integers are also not well defined but most implementations restrict integers to int32. You basically cannot send 64-bit integers without encoding them first. It gets really hairy.
For jsonrpc, there's no standard way to enumerate methods like there is in xml-rpc and jsonrpc isn't self-describing. Having an RPC be self-describing is really useful when dealing with a dynamic language because you can then avoid IDL compilation.
So as far as RPCs go, jsonrpc is actually a pretty bad one. The odd thing about it is you still have to parse json in JavaScript to avoid security problems. Parsing json in js seems to eliminate 99% of the useness of it.
Signature · 3 lines
Regards, Anthony Liguori
Quoted 4 lines · Anthony Liguori
Quoted 1 line · Avi Kivity
It looks brittle to me. What if you want to add an array? Some structure that contains a nested structure?
Can you give me concrete examples?
You want statistics for all network interfaces in one go. In RPC you return [ { name: blah stats: { rx: { packets: 123, bytes: 456, errors: 789 }, tx: etc. In a text protocol you have more work to write, parse, and most importantly document the protocol.
Quoted 3 lines
The typical way this would be handled in an RPC library in C is to introduce complex types. For instance, you could have a MonitorList and a corresponding
MON_LISTtype. FWIW, is there a jsonrpc library for C that's actually maintained?I can't find one that even has a functioning website.
I think a json C library would be sufficient, there likely are several. The RPC part is really trivial for a server.
Quoted 3 lines
Quoted 1 line
This is deep into overkill territory IMO. We should pick some format that is good enough and has implementations for a variety of languages. Luiz' RFC was well done, but the subject is complex enough to be difficult to get right. And as far as I could tell, all the issues are addressed by jsonrpc. The only downside I can see is that jsonrpc is some kind of ajax-kiddie web-2.0 script dom thingie while we are a serious UNIX style daemon that complies with RFC 123456.
There are many reasons why JSON is a bad representation. For starters, there is no way to encode NULL values in a JSON string. This means you cannot send binary data unless you base64 encode it first. The bounds of integers are also not well defined but most implementations restrict integers to int32. You basically cannot send 64-bit integers without encoding them first. It gets really hairy.
These look like show stoppers to me.
Quoted 1 line
For jsonrpc, there's no standard way to enumerate methods like there is in xml-rpc and jsonrpc isn't self-describing. Having an RPC be self-describing is really useful when dealing with a dynamic language because you can then avoid IDL compilation.
I think you can avoid IDL compilation even without a self describing RPC.
Quoted 1 line
So as far as RPCs go, jsonrpc is actually a pretty bad one. The odd thing about it is you still have to parse json in JavaScript to avoid security problems. Parsing json in js seems to eliminate 99% of the useness of it.
Are you referring to the fact that eval() on json is insecure? We don't have eval() in C.
Well, if not json, let's find some established RPC (or just data serialization) that is not XML-RPC or Sun RPC and use it, so we can concentrate on adding commands easily rather than writing formats, formatters, and parsers.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 1 line · Anthony Liguori
For starters, there is no way to encode NULL values in a JSON string. This means you cannot send binary data unless you base64 encode it first.
That seems to have been solved:
js> a = 'x\0y'
xy
js> a.length
3
I don't know enough javascript to verify that it's indeed a NUL, but I'm guessing that it is.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 6 lines · Avi Kivity
Quoted 1 line · Anthony Liguori
For starters, there is no way to encode NULL values in a JSON string. This means you cannot send binary data unless you base64 encode it first.
That seems to have been solved:
Quoted 6 lines
js> a = 'x\0y'
xy
js> a.length
3I don't know enough javascript to verify that it's indeed a NUL, but I'm guessing that it is.
This is a browser restriction.
Signature · 3 lines
Regards, Anthony Liguori
Quoted 7 lines · Anthony Liguori
Quoted 6 lines · Avi Kivity
Quoted 1 line · Anthony Liguori
For starters, there is no way to encode NULL values in a JSON string. This means you cannot send binary data unless you base64 encode it first.
That seems to have been solved:
Quoted 9 lines
Quoted 6 lines
js> a = 'x\0y'
xy
js> a.length
3I don't know enough javascript to verify that it's indeed a NUL, but I'm guessing that it is.
This is a browser restriction.
So this will only hit us is we port qemu to javascript and run it in a web broswer.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 1 line · Avi Kivity
So this will only hit us is we port qemu to javascript and run it in a web broswer.
You assume that client implementations are going to always accept nulls in strings. Considering that the most popular clients (i.e. browsers) don't support this, I don't think it's a safe assumption.
Regards,
Anthony Liguori
Signature · 3 lines
Regards, Anthony Liguori
Quoted 11 lines · Anthony Liguori
Quoted 2 lines · Avi Kivity
It looks brittle to me. What if you want to add an array? Some structure that contains a nested structure?
Can you give me concrete examples?
The typical way this would be handled in an RPC library in C is to introduce complex types. For instance, you could have a MonitorList and a corresponding
MON_LISTtype. FWIW, is there a jsonrpc library for C that's actually maintained?
this one is mentioned on the website and has been changed recently (may 2009):
http://fara.cs.uni-potsdam.de/~jsg/json_parser/
Quoted 6 lines
There are many reasons why JSON is a bad representation. For starters, there is no way to encode NULL values in a JSON string. This means you cannot send binary data unless you base64 encode it first. The bounds of integers are also not well defined but most implementations restrict integers to int32. You basically cannot send 64-bit integers without encoding them first. It gets really hairy.
i'm not sure why do you think that, but the JSON string are completly escaped and there's no restricted value (compared to xml) since everything can be as a last measure be escaped as decimal \1234 .. in a case of a null \0000
for Int, they don't have any precision (infinite number of digits), so at the C level they should be represented as string of digits. So that some library represent the int as a C int, doesn't mean the protocol enforce that on you.
If that would be a problem and you couldn't change the implementation to do the right thing, you could represent your 64 bits int as a json string, so that the implementation on the other side represent it adequately.
but again the protocol can handle any length of integer, and there's no encoding issue because it's just like a human would write an integer of any precision (i.e. 123218478219742718)
Quoted 4 lines
For jsonrpc, there's no standard way to enumerate methods like there is in xml-rpc and jsonrpc isn't self-describing. Having an RPC be self-describing is really useful when dealing with a dynamic language because you can then avoid IDL compilation.
does QMP solve the problem here ?
Quoted 4 lines
So as far as RPCs go, jsonrpc is actually a pretty bad one. The odd thing about it is you still have to parse json in JavaScript to avoid security problems. Parsing json in js seems to eliminate 99% of the useness of it.
but C is not javascript. it's doesn't have eval, so your argument here is bogus. also JSon doesn't imply to have any dependancy whatsoever on javascript.
Signature · 1 line
Vincent
Quoted 8 lines · Vincent Hanquez
Quoted 6 lines · Anthony Liguori
Quoted 2 lines · Avi Kivity
It looks brittle to me. What if you want to add an array? Some structure that contains a nested structure?
Can you give me concrete examples?
Quoted 8 lines
Quoted 3 lines
The typical way this would be handled in an RPC library in C is to introduce complex types. For instance, you could have a MonitorList and a corresponding
MON_LISTtype. FWIW, is there a jsonrpc library for C that's actually maintained?this one is mentioned on the website and has been changed recently (may 2009):
That's not a jsonrpc library, that's just a json parser.
Signature · 3 lines
Regards, Anthony Liguori
Quoted 6 lines · Anthony Liguori
Quoted 4 lines
this one is mentioned on the website and has been changed recently (may 2009):
That's not a jsonrpc library, that's just a json parser.
but this is completly trivial at this point. jsonrpc is a json message put in a certain way.
now providing there was a maintained library out there, would you use it ? or do you have more concerns that were unanswered ?
Signature · 1 line
Vincent
Quoted 2 lines
now providing there was a maintained library out there, would you use it ? or do you have more concerns that were unanswered ?
I have exactly one concern. If Qemu grows an RPC protocol, it will be used by a large number of other programs to control qemu. This is a good thing. But it is important that such a system
1) either supports some sort of introspection so that the functions and their arguments can be divined by the external program without knowing which particular version of qemu it is talking to.
2) and/or supports an IDL so that the interface is properly documented in something other than an under-edited text file.
To this end I tentitatively suggest d-bus as a possible choice. It fills 1 and 2 nicely and also allows each module to have its own interfaces and objects. The IDL will also generate C bindings for both ends.
I should add I'm not a fan of d-bus, but it seems a better fit than JSON-RPC
James.
Quoted 11 lines · James
Quoted 2 lines
now providing there was a maintained library out there, would you use it ? or do you have more concerns that were unanswered ?
I have exactly one concern. If Qemu grows an RPC protocol, it will be used by a large number of other programs to control qemu. This is a good thing. But it is important that such a system
1) either supports some sort of introspection so that the functions and their arguments can be divined by the external program without knowing which particular version of qemu it is talking to.
Quoted 7 lines
2) and/or supports an IDL so that the interface is properly documented in something other than an under-edited text file.
To this end I tentitatively suggest d-bus as a possible choice. It fills 1 and 2 nicely and also allows each module to have its own interfaces and objects. The IDL will also generate C bindings for both ends.
It depends what you mean by 'dbus' here. I don't think managing QEMU over the dbus bus provides the right architecture - I don't think you want every QEMU appearing on the bus.5B You could do direct peer-to-peer comms so you're just using libdbus for message encapsulation / processing, but there's not really much above XDR. XDR is nice because its a portable library that works on any OS, including OS-X and WIndows, and solely concerns itself with data encoding, not data transport.
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Quoted 7 lines
It depends what you mean by 'dbus' here. I don't think managing QEMU over the dbus bus provides the right architecture - I don't think you want every QEMU appearing on the bus. 5B You could do direct peer-to-peer comms so you're just using libdbus for message encapsulation / processing, but there's not really much above XDR. XDR is nice because its a portable library that works on any OS, including OS-X and WIndows, and solely concerns itself with data encoding, not data transport.
That leaves the rendezvous, and security issues to be re-invented wheels, you could use SUNRPC and XDR. That has an excellent IDL.
I don't see why you wouldn't want it on the 'bus', and more than you'd want it bound to a tcp port on localhost. It would make things a lot simpler: something like a little command line utility to connect a CDROM or ISO image to a Qemu instance which would be identified by some uuid or name would be very easy to implement.
James.
Quoted 8 lines · James
Quoted 7 lines
It depends what you mean by 'dbus' here. I don't think managing QEMU over the dbus bus provides the right architecture - I don't think you want every QEMU appearing on the bus. 5B You could do direct peer-to-peer comms so you're just using libdbus for message encapsulation / processing, but there's not really much above XDR. XDR is nice because its a portable library that works on any OS, including OS-X and WIndows, and solely concerns itself with data encoding, not data transport.
Quoted 8 lines
That leaves the rendezvous, and security issues to be re-invented wheels, you could use SUNRPC and XDR. That has an excellent IDL.
I don't see why you wouldn't want it on the 'bus', and more than you'd want it bound to a tcp port on localhost. It would make things a lot simpler: something like a little command line utility to connect a CDROM or ISO image to a Qemu instance which would be identified by some uuid or name would be very easy to implement.
This is exactly why I don't want it on the bus. I don't think the monitor interface is the general end-developer/user administrative interface. It is a control channel by which a mgmt tool will control QEMU, with the mgmt tool providing the end-developer/user admin interface. If someone wants to write a management app that controls QEMU instances and exposes an API on dbus that's fine, but the individual QEMU instances should not be directly exposed for use.
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Quoted 17 lines · Daniel P. Berrange
Quoted 7 lines · James
Quoted 7 lines
It depends what you mean by 'dbus' here. I don't think managing QEMU over the dbus bus provides the right architecture - I don't think you want every QEMU appearing on the bus. 5B You could do direct peer-to-peer comms so you're just using libdbus for message encapsulation / processing, but there's not really much above XDR. XDR is nice because its a portable library that works on any OS, including OS-X and WIndows, and solely concerns itself with data encoding, not data transport.
Quoted 8 lines
That leaves the rendezvous, and security issues to be re-invented wheels, you could use SUNRPC and XDR. That has an excellent IDL.
I don't see why you wouldn't want it on the 'bus', and more than you'd want it bound to a tcp port on localhost. It would make things a lot simpler: something like a little command line utility to connect a CDROM or ISO image to a Qemu instance which would be identified by some uuid or name would be very easy to implement.
Quoted 8 lines
This is exactly why I don't want it on the bus. I don't think the monitor interface is the general end-developer/user administrative interface. It is a control channel by which a mgmt tool will control QEMU, with the mgmt tool providing the end-developer/user admin interface. If someone wants to write a management app that controls QEMU instances and exposes an API on dbus that's fine, but the individual QEMU instances should not be directly exposed for use.
Why qemu shouldn't provide a monitor interface on dbus? The mgmt tool could still control qemu and provide the end-developer/user admin interface anyway.
Quoted 7 lines · Vincent Hanquez
Quoted 5 lines · Anthony Liguori
Quoted 5 lines
this one is mentioned on the website and has been changed recently (may 2009):
Quoted 8 lines
Quoted 2 lines
That's not a jsonrpc library, that's just a json parser.
but this is completly trivial at this point. jsonrpc is a json message put in a certain way.
now providing there was a maintained library out there, would you use it ? or do you have more concerns that were unanswered ?
There are two questions to resolve. The first is whether we should continue with the current direction (line-based protocol) or whether we should switch to an RPC. The second question is which RPC we should use.
I'm not at all convinced that we should switch to an RPC mechanism in the first place. Perhaps someone could summarize the advantages of doing this because right now, I don't see many.
With respect to RPC choice, if we did go that route, I'd be very concerned about using jsonrpc verses a more well established rpc. I would honestly prefer xml-rpc over jsonrpc.
One reason to choose an RPC is based on the adoption of it. You want to use something that has a vibrant community with well established client libraries to make writing clients as easy as possible. Without an active jsonrpc C library, it's hard to argue that jsonrpc has that. xml-rpc certainly does.
Regards,
Anthony Liguori
Quoted 19 lines · Anthony Liguori
Quoted 11 lines · Vincent Hanquez
Quoted 8 lines · Anthony Liguori
Quoted 6 lines
this one is mentioned on the website and has been changed recently (may 2009):
That's not a jsonrpc library, that's just a json parser.
Quoted 6 lines
but this is completly trivial at this point. jsonrpc is a json message put in a
certain way.now providing there was a maintained library out there, would you use it ? or do you have more concerns that were unanswered ?
Quoted 7 lines
There are two questions to resolve. The first is whether we should continue with the current direction (line-based protocol) or whether we should switch to an RPC. The second question is which RPC we should use.
I'm not at all convinced that we should switch to an RPC mechanism in the first place. Perhaps someone could summarize the advantages of doing this because right now, I don't see many.
It really depends what you consider the purpose of the monitor to be. If the monitor is the "API" application developers use for management, then a full blown RPC mechaism makes sense. If the monitor is an internal control mechanism, to enable the building mgmt tools, then RPC is overkill. I consider QEMU's monitor to be the latter, and interface for a single application to use (and developer debugging). If we try to turn the monitor into the general end-app developer mgmt interface, then you open the floodgates to vastly more functionality being pushed into the QEMU codebase. I don't think we want that, QEMU should be lean & mean, and if you want fancy RPC systems then do it in a tool layered above, be it libvirt, XenD, a CIM agent, or something else.
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Quoted 6 lines · Daniel P. Berrange
It really depends what you consider the purpose of the monitor to be. If the monitor is the "API" application developers use for management, then a full blown RPC mechaism makes sense. If the monitor is an internal control mechanism, to enable the building mgmt tools, then RPC is overkill.
I don't understand the difference. Every controllable feature should be exposed by the monitor, and nothing else. In both cases.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 3 lines · Anthony Liguori
There are two questions to resolve. The first is whether we should continue with the current direction (line-based protocol) or whether we should switch to an RPC. The second question is which RPC we should use.
I'm not at all convinced that we should switch to an RPC mechanism in the first place. Perhaps someone could summarize the advantages of doing this because right now, I don't see many.
Less effort for us.
Less effort for clients.
Less documentation effort.
Less likelihood of design and implementation holes (cf. UTF-8, quoting).
Richer data types (arrays, structures, nesting).
Quoted 1 line
With respect to RPC choice, if we did go that route, I'd be very concerned about using jsonrpc verses a more well established rpc. I would honestly prefer xml-rpc over jsonrpc.
I agree xml-rpc is a more rational choice than jsonrpc, but I cannot find it in my heart to say something nice about xml. Additionally, XML parsers are pretty heavy.
Quoted 1 line
One reason to choose an RPC is based on the adoption of it. You want to use something that has a vibrant community with well established client libraries to make writing clients as easy as possible. Without an active jsonrpc C library, it's hard to argue that jsonrpc has that. xml-rpc certainly does.
jsonrpc really is a trivial addition over json, and json is pretty widespread. I think I saw a Python jsonrpc implementation.
The important thing however is to reuse, it doesn't really matter what we reuse really as long as it's good enough.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 7 lines · Avi Kivity
Quoted 3 lines
With respect to RPC choice, if we did go that route, I'd be very concerned about using jsonrpc verses a more well established rpc. I would honestly prefer xml-rpc over jsonrpc.
I agree xml-rpc is a more rational choice than jsonrpc, but I cannot find it in my heart to say something nice about xml. Additionally, XML parsers are pretty heavy.
I'm not defending XML here, but:
You can code a minimal XML parser in straight C quite easily, if it's a restricted subset.
That is, if you specify the XML won't use any of the irrelevant features like PIs, DTDs, namespaces and named entities and know it's all UTF-8. Maybe even no comments. Then it's just nested elements, attributes and strings, with < to escape a few characters.
But the XML-RPC spec does not appear to restrict the type of XML, so you have to handle every XML feature to be XML-RPC compliant, unfortunately.
Some other "wire" protocols do specify a reduced subset of XML, which means you don't have to use a heavyweight library for them, which is nice.
I think feature-restricted XML is quite good for this sort of thing.
It's a bit more self-documenting than JSON. But for some reason, lots
of XML users use it very badly, doing stupid things like <array><element><type><typename>int</typename></type><subscript>0</subscript><element-value>0</element-value></element>...</array>. That's an exaggeration but I'm sure you all recognise the pattern.
XML and JSON both have the same ugly problem with binary data: they can't carry it. It's usually base64 encoded. Then again the QEMU monitor is no better this respect :-)
-- Jamie
Quoted 6 lines · Jamie Lokier
Quoted 4 lines · Avi Kivity
Quoted 4 lines
With respect to RPC choice, if we did go that route, I'd be very concerned about using jsonrpc verses a more well established rpc. I would honestly prefer xml-rpc over jsonrpc.
Quoted 8 lines
Quoted 4 lines
I agree xml-rpc is a more rational choice than jsonrpc, but I cannot find it in my heart to say something nice about xml. Additionally, XML parsers are pretty heavy.
I'm not defending XML here, but:
You can code a minimal XML parser in straight C quite easily,
One of the points of using an existing RPC implementation is not to have to code it up.
Quoted 2 lines
if it's a restricted subset.
You can't control what's coming in from the other side.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 12 lines · Avi Kivity
Quoted 11 lines · Jamie Lokier
Quoted 8 lines · Avi Kivity
Quoted 4 lines
With respect to RPC choice, if we did go that route, I'd be very concerned about using jsonrpc verses a more well established rpc. I would honestly prefer xml-rpc over jsonrpc.
I agree xml-rpc is a more rational choice than jsonrpc, but I cannot find it in my heart to say something nice about xml. Additionally, XML parsers are pretty heavy.
Quoted 6 lines
Quoted 3 lines
I'm not defending XML here, but:
You can code a minimal XML parser in straight C quite easily,
One of the points of using an existing RPC implementation is not to have to code it up.
In my experience, using some existing RPC implementations tends to involve quite a lot of code, and implementing from scratch only a little more, when we're talking about *simple* RPC protocols. Like QEMU's monitor for example. You still reuse code of course, but it's things like sockets, buffering, select and dispatch loops.
Obviously if it's something larger it's good to start with a good library. One which not only exists, but doesn't need a lot of application boilerplate. RPC libraries vary a lot in how much application code they need you to write.
I don't know whether there are good XML-RPC libraries for C. There doesn't seem to be much for JSON-RPC services.
Quoted 3 lines
Quoted 1 line
if it's a restricted subset.
You can't control what's coming in from the other side.
You can reject it if the protocol specification says it's not allowed.
Some protocols do that. Otherwise you have to process it correctly. It's trivial to reject byte values that shouldn't be there.
-- Jamie
Quoted 2 lines · Jamie Lokier
You can code a minimal XML parser in straight C quite easily, if it's a restricted subset.
even the restricted subset is not as straighforward as a json parser. and usually using a subset means you can't interact correctly with the one that does the full spec.
Quoted 3 lines
XML and JSON both have the same ugly problem with binary data: they can't carry it. It's usually base64 encoded. Then again the QEMU monitor is no better this respect :-)
JSon ***DOES*** do binary data.
C String "abc\0\xff" -> Json String "abc\0000\00ff"
Signature · 1 line
Vincent
Quoted 8 lines · Vincent Hanquez
Quoted 2 lines · Jamie Lokier
You can code a minimal XML parser in straight C quite easily, if it's a restricted subset.
even the restricted subset is not as straighforward as a json parser. and usually using a subset means you can't interact correctly with the one that does the full spec.
Quoted 7 lines
Quoted 3 lines
XML and JSON both have the same ugly problem with binary data: they can't carry it. It's usually base64 encoded. Then again the QEMU monitor is no better this respect :-)
JSon ***DOES*** do binary data.
C String "abc\0\xff" -> Json String "abc\0000\00ff"
I find the Json representation problematic. In C you have two distinct data types - null-terminated string where the length is implicitly known from the content (char *string) and a binary data blob (char *buffer, int size). If you encode them into the same JSON data type and don't supply "out-of-band" information about which one of the C types is it, the receiver has no way to decide what to decode it into. JSONRPC allows supplying this "out-of-band" information only for the JSON data types which is very limiting.
For text based protocols it's vital to separate the syntax from semantics and decoding the above would require knowning the specific context and semantics.
A more natural representation of binary blob in JSON would be array of numbers, but that would have a big overhead.
Best regards,
Filip Navara
Quoted 9 lines · Vincent Hanquez
Quoted 8 lines · Jamie Lokier
Quoted 2 lines
You can code a minimal XML parser in straight C quite easily, if it's a restricted subset.
even the restricted subset is not as straighforward as a json parser. and usually using a subset means you can't interact correctly with the one that does the full spec.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
XML and JSON both have the same ugly problem with binary data: they can't carry it. It's usually base64 encoded. Then again the QEMU monitor is no better this respect :-)
JSon ***DOES*** do binary data.
C String "abc\0\xff" -> Json String "abc\0000\00ff"
Any reason you can't simply put the UTF-8 encoding of U+0000 and U+00FF directly in the string?
(Let's ignore Java, which encodes U+0000 in "Java-UTF-8" differently from everyone else's UTF-8!)
Quoted 9 lines · Filip Navara
I find the Json representation problematic. In C you have two distinct data types - null-terminated string where the length is implicitly known from the content (char *string) and a binary data blob (char *buffer, int size). If you encode them into the same JSON data type and don't supply "out-of-band" information about which one of the C types is it, the receiver has no way to decide what to decode it into. JSONRPC allows supplying this "out-of-band" information only for the JSON data types which is very limiting.
Quoted 6 lines
For text based protocols it's vital to separate the syntax from semantics and decoding the above would require knowning the specific context and semantics.
A more natural representation of binary blob in JSON would be array of numbers, but that would have a big overhead.
Actually, an array doesn't add much more overhead :-)
Binary has a big overhead in JSON as a string.
Encode the blob [0,1,2,3,252,253,254,255]:
JSON: "\0\1\2\3\xfc\xfd\xfe\xff"
That's fine if you've just got a few non-ASCII bytes. But for general binary, base64 is much more compact.
Base64 expands by about 4/3, whereas a JSON \x-encoded string expands by up to 4 times. Even a hex string is more compact :-)
A further complication is that a JSON string carries Unicode (which is good for text), so at some point you have to know to signal a parse error when any characters are outside the range 0-255.
But I don't know why we're talking about binary, as the monitor (old or proposed) doesn't handle binary either, and no particular need for binary has come up...
-- Jamie
Quoted 9 lines · Filip Navara
Quoted 8 lines · Vincent Hanquez
Quoted 2 lines · Jamie Lokier
You can code a minimal XML parser in straight C quite easily, if it's a restricted subset.
even the restricted subset is not as straighforward as a json parser. and usually using a subset means you can't interact correctly with the one that does the full spec.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
XML and JSON both have the same ugly problem with binary data: they can't carry it. It's usually base64 encoded. Then again the QEMU monitor is no better this respect :-)
JSon ***DOES*** do binary data.
C String "abc\0\xff" -> Json String "abc\0000\00ff"
btw, sorry i meant \\ instead of \ in the json string. as in: 'a' 'b' 'c' '\\' '0' '0' '0' '0' '\\' '0' '0' 'f' 'f'
Quoted 8 lines
I find the Json representation problematic. In C you have two distinct data types - null-terminated string where the length is implicitly known from the content (char *string) and a binary data blob (char *buffer, int size). If you encode them into the same JSON data type and don't supply "out-of-band" information about which one of the C types is it, the receiver has no way to decide what to decode it into. JSONRPC allows supplying this "out-of-band" information only for the JSON data types which is very limiting.
it's a problem related to the representation of your string, not of JSon.
in most case it doesn't matter though, since if your string contains null that you want to send you need to use the correct "json_append_string" in the first place which will take the lenght as parameter and on the receiving end you can expect that every characters received were meant to be sent and need to use the proper accessor to get the string of the proper lenght.
Signature · 1 line
Vincent
Quoted 30 lines · Vincent Hanquez
Quoted 9 lines · Filip Navara
Quoted 8 lines · Vincent Hanquez
Quoted 2 lines · Jamie Lokier
You can code a minimal XML parser in straight C quite easily, if it's a restricted subset.
even the restricted subset is not as straighforward as a json parser. and usually using a subset means you can't interact correctly with the one that does the full spec.
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
XML and JSON both have the same ugly problem with binary data: they can't carry it. It's usually base64 encoded. Then again the QEMU monitor is no better this respect :-)
JSon ***DOES*** do binary data.
C String "abc\0\xff" -> Json String "abc\0000\00ff"
btw, sorry i meant \\ instead of \ in the json string. as in: 'a' 'b' 'c' '\\' '0' '0' '0' '0' '\\' '0' '0' 'f' 'f'
Quoted 8 lines
I find the Json representation problematic. In C you have two distinct data types - null-terminated string where the length is implicitly known from the content (char *string) and a binary data blob (char *buffer, int size). If you encode them into the same JSON data type and don't supply "out-of-band" information about which one of the C types is it, the receiver has no way to decide what to decode it into. JSONRPC allows supplying this "out-of-band" information only for the JSON data types which is very limiting.
Quoted 8 lines
it's a problem related to the representation of your string, not of JSon.
in most case it doesn't matter though, since if your string contains null that you want to send you need to use the correct "
json_append_string" in the first place which will take the lenght as parameter and on the receiving end you can expect that every characters received were meant to be sent and need to use the proper accessor to get the string of the proper lenght.
*need to use the proper accessor to get the string of the proper length*
This is precisely what I am talking about when I'm talking about mixing syntax and semantics. You'd require the parser to know the semantics based on context. That's bad enough of itself, but fortunately JSONRPC supports "service descriptions" which allow the parser to be fed with some generic description of the semantics. Unfortunately this service description doesn't cover all C data types and can't even describe the basic integer sizes, which would allow for syntax and semantic validation of the RPC payload by the parser.
Best regards,
Filip Navara
Quoted 14 lines · Vincent Hanquez
Quoted 13 lines · Filip Navara
Quoted 12 lines · Vincent Hanquez
Quoted 2 lines · Jamie Lokier
You can code a minimal XML parser in straight C quite easily, if it's a restricted subset.
even the restricted subset is not as straighforward as a json parser. and usually using a subset means you can't interact correctly with the one that does the full spec.
Quoted 3 lines
XML and JSON both have the same ugly problem with binary data: they can't carry it. It's usually base64 encoded. Then again the QEMU monitor is no better this respect :-)
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
JSon ***DOES*** do binary data.
C String "abc\0\xff" -> Json String "abc\0000\00ff"
btw, sorry i meant \\ instead of \ in the json string. as in: 'a' 'b' 'c' '\\' '0' '0' '0' '0' '\\' '0' '0' 'f' 'f'
Wow. So a JSON string is actually _larger_ than an array of numbers, for some binary data?
As in:
"abc\\0000\\00ff"
versus
[97,98,99,0,255]
Heh :-)
-- Jamie
Anthony Liguori writes ("Re: [Qemu-devel] [PATCH 01/11] QMP: Introduce specification file"):
Quoted 3 lines
I'm not at all convinced that we should switch to an RPC mechanism in the first place. Perhaps someone could summarize the advantages of doing this because right now, I don't see many.
I'm keen on RPC-like systems. The main advantage is that you don't have to invent and write a whole lot of marshalling and parsing code; the RPC system will provide all that for you. The more sophisticated systems deal with message transport as well, so you don't need to mess about with sockets either.
Parsers and formatters are one of the most tedious bits of code to write and they are also critical for security, reliability and data transparency. It really doesn't make sense to roll your own for any nontrivial problem.
If you pick the right RPC system you can get, entirely for free (well, entirely piggybacking on other people's work) compact and efficiently parseable binary wire formats. Debugging trace output facilities, which produce in nice human readable formats, are often provided as a standard feature for no added effort. Bindings for a wide variety of programming languages can be easily found. If you pick a bus- or broker-based system (dbus, XDR, corba even) you get discovery and rendezvous for free too. Many systems have builtin facilities for cross-version compatibility, of varying sophistication.
I would go further and say that we should use an RPC system with a machine-readable IDL (Interface Description Language). A formal IDL provides an number of very significant advantages. If you use an IDL, the protocol is documented not primarily in a human-only-readable text file, but primarily in the IDL, which means that the documentation cannot get out of date. Introspection and debugging features are easier to provide because all of the programs have actual knowledge of the message format and structure. The wire formats can be much more compact and efficient, because they do not need to tag every value with its type and format - the receiving end knows what the format is supposed to be and can get straight to unpicking it.
You can have an argument about which specific RPC mechanism to use, but it's an embarrassment of riches compared to designing and writing your own textual protocol, particularly if that protocol needs to be quite complex and asynchronous.
I think dbus would be a good choice, but even XML-RPC (which, frankly, is flabby, overcomplex, slow, hard to read, etc.), JSON (which doesn't have an IDL that can even do checking), and ASN.1/DER/BER (which is a complexity nightmare) would be an improvement over no automatic parsing machinery.
Quoted 5 lines
One reason to choose an RPC is based on the adoption of it. You want to use something that has a vibrant community with well established client libraries to make writing clients as easy as possible. Without an active jsonrpc C library, it's hard to argue that jsonrpc has that. xml-rpc certainly does.
Yes.
Ian.
Quoted 5 lines · Ian Jackson
I think dbus would be a good choice, but even XML-RPC (which, frankly, is flabby, overcomplex, slow, hard to read, etc.), JSON (which doesn't have an IDL that can even do checking), and ASN.1/DER/BER (which is a complexity nightmare) would be an improvement over no automatic parsing machinery.
I strongly dislike D-Bus, having been forced to use it to manage Linux Bluetooth connections. (The old way with a nice traditional unix-style executable command to control them was far nicer and actually worked).
D-Bus is impossible to use properly from shell with the available shell client (some value types aren't supported), and you have to read application source code to figure out what calls to use, if the applications RPCs aren't well documented (which they aren't for Linux Bluetooth). There's no useful introspection.
D-Bus seems to be ok for asynchronous notifications, such as what's used to update icons on a desktop in response to background events. But it's not at all transactional so you often have temporary display inconsistencies in a GUI using it.
-- Jamie
Quoted 18 lines · Anthony Liguori
Quoted 17 lines · Vincent Hanquez
Quoted 9 lines · Anthony Liguori
Quoted 6 lines
this one is mentioned on the website and has been changed recently (may 2009):
That's not a jsonrpc library, that's just a json parser.
but this is completly trivial at this point. jsonrpc is a json message put in a
certain way.
Quoted 6 lines
Quoted 2 lines
now providing there was a maintained library out there, would you use it ? or do you have more concerns that were unanswered ?
There are two questions to resolve. The first is whether we should continue with the current direction (line-based protocol) or whether we should switch to an RPC. The second question is which RPC we should use.
I've spent the last 5 years working on different projects that involved text/line-based protocols. In fact my current day job involves working on a program that works with wide variety of such protocols (POP3, SMTP, IMAP, ACAP, HTTP to name a few) and based on my experience I would seriously recommend using an existing RPC protocol instead. While it's simple to design a synchronous protocol like the original POP3 it gets a lot harder with asynchronous protocols like IMAP. Even if you decide for a custom line-level protocol I'd suggest to learn from the past mistakes, more details below.
The current draft specification used a mix of semantics of the forementioned protocols - error handling based on POP3 (+OK/-ERR), asynchronous events based on IMAP (* prefix). IMAP-like tagging of commands was also suggested (to deal with asynchronous commands) and issues have been raised concerning multi-line responses. Note that all this gets a bit hairy when you combine it. Just a few issues that come to mind:
* An event happens while a multi-line response is currently written to the stream. The behavior has to be defined somehow - either by specifying that the multi-line response mustn't be interrupted by asynchronous events or that it can be interrupted, but only on line breaks. This issue don't arise with message-based protocols.
* Should the asynchronous replies (* EVENT) be tagged (ie. C: "<tag1> reboot", S: "* <tag1> reset") when they result from a command issued by the manager? If they are not tagged then it's impossible to find out what (if any) of the commands caused the event when multiple asynchronous commands are in progress. That's what basic IMAP does (as specified by RFC 3501). What IMAP did wrong [1] is that even the basic responses to commands like SEARCH are returned as untagged asynchronous notifications, which makes the client implementation hard. Several extensions were later developed to mitigate it and the protocol is now a messy combination of both tagged and untagged responses with no universal syntax. On the other hand, if you tag the events then it's hard for the server (in this case QEMU) to keep track of it and a special syntax still has to be present for truly asynchronous events.
* It is necessary to separate the syntax of the data protocol from the semantics of the commands itself. The syntax should be generic enough to support transferring all the necessary information (integers, strings, lists) and simple enough to be parsable even if you receive the data in chunks (assuming a text-based protocol). This is where IMAP went really wrong, since parsing the server responses requires knowning the semantics of ALL the commands.
* Many of you asked for facility to negotiate capabilities. I'd suggest the following - the server would lists capabilities by name as a list on connect and the client would be able to specifically enable those that change the semantics of commands. Example:
S: + OK QEMU 0.10.50 QMP 0.1 (pci-addr, x-some-extension)
C: enable pci-addr
S: + OK pci-addr extension enabled
The "enable" command would optionally allow for a list to be specified, to enable more extensions at once.
Best regards,
Filip Navara
[1] ...according to large amount of people with the exception of Mark Crispin.
Quoted 7 lines · On Wed, 24 Jun 2009 16:03:13 +0200 Filip Navara <[email protected]> wrote:
The current draft specification used a mix of semantics of the forementioned protocols - error handling based on POP3 (+OK/-ERR), asynchronous events based on IMAP (* prefix). IMAP-like tagging of commands was also suggested (to deal with asynchronous commands) and issues have been raised concerning multi-line responses. Note that all this gets a bit hairy when you combine it. Just a few issues that come to mind:
Although I think QMP is very close to what QEMU needs, there is no
doubt it's not complete.
I appreciate your feedback, but I will only go through it if we
decide using QMP.
Quoted 3 lines · Anthony Liguori
There are two questions to resolve. The first is whether we should continue with the current direction (line-based protocol) or whether we should switch to an RPC. The second question is which RPC we should use.
well, when choosing simple RPC like xmlrpc or jsonrpc you still need to do the transport. so it could well be a line-based protocol that happens to contains query in a JSONRPC or XMLRPC format ... as long as you don't pretty print JSONRPC or XMLRPC (guaranteed to not contain \n). a bit clunky but possible.
Quoted 7 lines
I'm not at all convinced that we should switch to an RPC mechanism in the first place. Perhaps someone could summarize the advantages of doing this because right now, I don't see many.
With respect to RPC choice, if we did go that route, I'd be very concerned about using jsonrpc verses a more well established rpc. I would honestly prefer xml-rpc over jsonrpc.
while i have a profound distaste of xmlrpc ... i still think that would be a improvement against the current situation .. or even to QMP.
however you have to keep in mind that xmlrpc means no binary data, and depending on an XML library (compare to beeing able to pull a very tiny code base directly in qemu tree)
Quoted 5 lines
One reason to choose an RPC is based on the adoption of it. You want to use something that has a vibrant community with well established client libraries to make writing clients as easy as possible. Without an active jsonrpc C library, it's hard to argue that jsonrpc has that. xml-rpc certainly does.
it might be missing for C, but lots of language got a maintained and used library nowadays. We're using it in ocaml, and i have to say we haven't been disappointed using it, compared to the number of disappointments with xmlrpc (and mostly xml in general).
That said, I'll make a C library even if qemu doesn't end up using it.
Signature · 1 line
Vincent
Quoted 16 lines · Anthony Liguori
Quoted 7 lines · Vincent Hanquez
Quoted 5 lines · Anthony Liguori
Quoted 5 lines
this one is mentioned on the website and has been changed recently (may 2009):
Quoted 7 lines
Quoted 2 lines
That's not a jsonrpc library, that's just a json parser.
but this is completly trivial at this point. jsonrpc is a json message put in a certain way.
now providing there was a maintained library out there, would you use it ? or do you have more concerns that were unanswered ?
Quoted 8 lines
There are two questions to resolve. The first is whether we should continue with the current direction (line-based protocol) or whether we should switch to an RPC. The second question is which RPC we should use.
I'm not at all convinced that we should switch to an RPC mechanism in the first place. Perhaps someone could summarize the advantages of doing this because right now, I don't see many.
Using an IDL it would be easier to generate boilerplate code for different languages, easier to document and evolve the interface. If the rpc library is good enough we would get marshalling, data transport, rendezvous, introspection and security already solved for us.
Quoted 11 lines · Anthony Liguori
There are two questions to resolve. The first is whether we should continue with the current direction (line-based protocol) or whether we should switch to an RPC. The second question is which RPC we should use.
I'm not at all convinced that we should switch to an RPC mechanism in the first place. Perhaps someone could summarize the advantages of doing this because right now, I don't see many.
With respect to RPC choice, if we did go that route, I'd be very concerned about using jsonrpc verses a more well established rpc. I would honestly prefer xml-rpc over jsonrpc.
We are an example of a end user of qemu (or more specifically qemu-kvm) that doesn't go via a management layer like libvirt. Our management shell scripts directly control the virtual machines using the current line-oriented 'human friendly' monitor. This is a bit of a pain, but not too bad in practice.
A more regular and well defined line-based protocol would be a big plus for us, whereas something like jsonrpc or xml-rpc would be a complete disaster to call from the shell---less useful than the current human oriented monitor rather than more so.
(Aside from the practical question, personally I share your gut feeling that it seems like a sledgehammer to crack a nut and not a good fit at all for this layer of the stack.)
Cheers,
Chris.
Quoted 10 lines · Chris Webb
Quoted 8 lines · Anthony Liguori
There are two questions to resolve. The first is whether we should continue with the current direction (line-based protocol) or whether we should switch to an RPC. The second question is which RPC we should use.
I'm not at all convinced that we should switch to an RPC mechanism in the first place. Perhaps someone could summarize the advantages of doing this because right now, I don't see many.
Quoted 8 lines
Quoted 3 lines
With respect to RPC choice, if we did go that route, I'd be very concerned about using jsonrpc verses a more well established rpc. I would honestly prefer xml-rpc over jsonrpc.
We are an example of a end user of qemu (or more specifically qemu-kvm) that doesn't go via a management layer like libvirt. Our management shell scripts directly control the virtual machines using the current line-oriented 'human friendly' monitor. This is a bit of a pain, but not too bad in practice.
I do the same. I even have a QEMU-monitor-multiplexor script, which accepts multiple monitor connections and issues the commands over the real monitor. So that I can have one script managing (basically insert/remove disk images, trigger reboots etc.), while still having monitor accesss from the command line for those special extras.
Quoted 4 lines
A more regular and well defined line-based protocol would be a big plus for us, whereas something like jsonrpc or xml-rpc would be a complete disaster to call from the shell---less useful than the current human oriented monitor rather than more so.
It's fine as long as there's a good shell-friendly utility.
Unfortunately my experience of using D-Bus through it's shell utility was not positive. You need the utility to do more than just pass the whole request and response back and forth. It needs to combine and extract fields in compound structures, and escape/unescape when you need that.
-- Jamie
On Wed, Jun 24, 2009 at 2:46 PM, Anthony Liguori<[email protected]> wrote:
[snip]
Quoted 9 lines
With respect to RPC choice, if we did go that route, I'd be very concerned about using jsonrpc verses a more well established rpc. I would honestly prefer xml-rpc over jsonrpc.
One reason to choose an RPC is based on the adoption of it. You want to use something that has a vibrant community with well established client libraries to make writing clients as easy as possible. Without an active jsonrpc C library, it's hard to argue that jsonrpc has that. xml-rpc certainly does.
After reading the XDR specification I realized that the basic blocks are already implemented in QEMU.
Data type mapping:
Integer <-> qemu_put_sbe32 / qemu_get_sbe32
Unsigned Integer <-> qemu_put_be32 / qemu_get_be32 Enumeration - same as signed integer
Boolean - same as signed interger, limited to values 0 and 1 Hyper Integer <-> qemu_put_sbe64 / qemu_get_sbe64 Unsigned Hyper Integer <-> qemu_put_be64 / qemu_get_be64 Floating-point, Double-precision Floating-point, Quadruple-precision Floating-point <-> ? (uses IEEE notation)
Fixed-length Opaque Data <->
void qemu_put_xdr_buffer(QEMUFile *f, const uint8_t *buf, int size)
{
int reminder = size % 4;
qemu_put_buffer(f, buf, size);
while (--reminder >= 0)
qemu_put_byte(f, 0);
}Variable-length Opaque Data <->
void qemu_put_xdr_var_buffer(QEMUFile *f, const uint8_t *buf, int size)
{
int reminder = size % 4;
qemu_put_be32(f, size);
qemu_put_buffer(f, buf, size);
while (--reminder >= 0)
qemu_put_byte(f, 0);
}String <-> same as qemu_put_xdr_var_buffer, ASCII bytes (*not characters*, the encoding of characters is not part of the RFC, NFSv4 uses UTF-8 for example), null-terminator is not sent
Fixed-length Array <-> for loop with the elements described using the basic types above
Variable-length Array <-> qemu_put_be32 + for loop / qemu_get_be32 + allocation + for loop
Structure, Discriminated Union <-> built from the basic data types, same as in C Void <-> nothing :)
Given the fact that we practically have the library for XDR it would be very easy to build Sun RPC server on top of it. So my vote is for SunRPC.
Best regards,
Filip Navara
Quoted 4 lines · Filip Navara
Given the fact that we practically have the library for XDR it would be very easy to build Sun RPC server on top of it. So my vote is for SunRPC.
My experience with SunRPC (NFS) has been pretty bad. It may have been due to implementation details, not to the spec itself, but it definitely left a bad taste in my mouth. Also, I don't see a SunRPC implementation for Python, which (along with similar languages) is an ideal platform for controlling qemu. I'd rather have something more modern.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Avi Kivity
Quoted 5 lines · Filip Navara
Given the fact that we practically have the library for XDR it would be very easy to build Sun RPC server on top of it. So my vote is for SunRPC.
My experience with SunRPC (NFS) has been pretty bad. It may have been due to implementation details, not to the spec itself, but it definitely left a bad taste in my mouth.
I have no experience with it, but the specifications looked simple and sane.
Quoted 3 lines
Also, I don't see a SunRPC implementation for
Python, which (along with similar languages) is an ideal platform for controlling qemu. I'd rather have something more modern.
http://svn.python.org/projects/python/trunk/Demo/rpc/
http://www.cs.umd.edu/~gaburici/rpc/
Libraries are available for vast amount of languages, including Java and .NET.
Best regards,
Filip Navara
Quoted 11 lines · Avi Kivity
Quoted 4 lines · Filip Navara
Given the fact that we practically have the library for XDR it would be very easy to build Sun RPC server on top of it. So my vote is for SunRPC.
My experience with SunRPC (NFS) has been pretty bad. It may have been due to implementation details, not to the spec itself, but it definitely left a bad taste in my mouth. Also, I don't see a SunRPC implementation for Python, which (along with similar languages) is an ideal platform for controlling qemu. I'd rather have something more modern.
SunRPC works, and it's fine if you always access it through utilities, like say a C command-line client. Hiding the SunRPC aspect :-)
It's pretty fast too, so good for things like network filesystems.
But it's a real pain to use from scripts if you don't have a pre-built utility which implements all application RPC functions already.
The basic problem is needing to create and compile C code - for all clients as well as the server - each time an RPC function is added or changed.
QEMU's monitor is already much easier to use than that.
When a new monitor command is added, you can use it from scripts as trivial one-liner text commands, assuming you already have a way to connect to the monitor.
-- Jamie
Quoted 13 lines · Jamie Lokier
Quoted 11 lines · Avi Kivity
Quoted 4 lines · Filip Navara
Given the fact that we practically have the library for XDR it would be very easy to build Sun RPC server on top of it. So my vote is for SunRPC.
My experience with SunRPC (NFS) has been pretty bad. It may have been due to implementation details, not to the spec itself, but it definitely left a bad taste in my mouth. Also, I don't see a SunRPC implementation for Python, which (along with similar languages) is an ideal platform for controlling qemu. I'd rather have something more modern.
Quoted 7 lines
SunRPC works, and it's fine if you always access it through utilities, like say a C command-line client. Hiding the SunRPC aspect :-)
It's pretty fast too, so good for things like network filesystems.
But it's a real pain to use from scripts if you don't have a pre-built utility which implements all application RPC functions already.
True. Text-based RPCs as JSON-RPC and XML-RPC are half as painful, but still painful.
Quoted 3 lines
The basic problem is needing to create and compile C code - for all clients as well as the server - each time an RPC function is added or changed.
You don't need rebuild the client if the server interface stays compatible and if you don't need the new functionality. Also my initial idea was to use QEMU functions for building the XDR binary payloads inside QEMU. Not sure if it's good idea, but it would work and it's easy to implement for the proposed monitor_print_data function.
Quoted 1 line
QEMU's monitor is already much easier to use than that.
For human? Definitely! For shell scripts? Yes. For libraries that need to parse the data reliably? No.
Quoted 3 lines
When a new monitor command is added, you can use it from scripts as trivial one-liner text commands, assuming you already have a way to connect to the monitor.
Since when was calling from scripts added as requirement? Don't get me wrong, I would be happy to cover this use case as well. It's a diverse use case from the libvirt usage though, in my humble opinion. I'd prefer killing one bird to knocking two of them and leaving them half-dead.
Best regards,
Filip Navara
Quoted 4 lines · Filip Navara
Quoted 1 line
QEMU's monitor is already much easier to use than that.
For human? Definitely! For shell scripts? Yes. For libraries that need to parse the data reliably? No.
Yes, it could do with a more well-defined formatting.
Documented escaping of unusual characters in names would be a start :-)
Quoted 7 lines
Quoted 3 lines
When a new monitor command is added, you can use it from scripts as trivial one-liner text commands, assuming you already have a way to connect to the monitor.
Since when was calling from scripts added as requirement? Don't get me wrong, I would be happy to cover this use case as well. It's a diverse use case from the libvirt usage though, in my humble opinion.
Aren't most QEMU management programs (other than libvirt) scripts?
Looks to me like "works with libvirt and other management programs" implies that you can use it from scripts, because many management programs are, in fact, scripts.
If the requirement is simply "works with libvirt", then closer integration with libvirt would be better than trying to cleanly RPC everything. After all, libvirt already has it's own RPC layer for everything connecting to it, and libvirt people want all QEMU interaction to happen via libvirt anyway. There's no point having multiple different RPC layers carrying exactly the same commands in different ways.
-- Jamie
Quoted 14 lines · Jamie Lokier
Quoted 4 lines · Filip Navara
Quoted 1 line
QEMU's monitor is already much easier to use than that.
For human? Definitely! For shell scripts? Yes. For libraries that need to parse the data reliably? No.
Yes, it could do with a more well-defined formatting.
Documented escaping of unusual characters in names would be a start :-)
Quoted 4 lines
Quoted 3 lines
When a new monitor command is added, you can use it from scripts as trivial one-liner text commands, assuming you already have a way to connect to the monitor.
Quoted 5 lines
Quoted 3 lines
Since when was calling from scripts added as requirement? Don't get me wrong, I would be happy to cover this use case as well. It's a diverse use case from the libvirt usage though, in my humble opinion.
Aren't most QEMU management programs (other than libvirt) scripts?
I don't feel competent to answer that.
Quoted 3 lines
Looks to me like "works with libvirt and other management programs" implies that you can use it from scripts, because many management programs are, in fact, scripts.
If that was true then certainly a text/line-based protocol would make sense (possibly in addition to a simple RPC one, if desired). The thing I am worried about is that several corner cases are currently not defined in the proposed protocol and it introduces asynchronous events, which is actually harder to get right in shell scripts than the human protocol.
Quoted 10 lines
If the requirement is simply "works with libvirt", then closer integration with libvirt would be better than trying to cleanly RPC everything. After all, libvirt already has it's own RPC layer for everything connecting to it, and libvirt people want all QEMU interaction to happen via libvirt anyway. There's no point having multiple different RPC layers carrying exactly the same commands in different ways.
-- Jamie
Best regards,
Filip Navara
Quoted 10 lines · Filip Navara
Quoted 3 lines
Looks to me like "works with libvirt and other management programs" implies that you can use it from scripts, because many management programs are, in fact, scripts.
If that was true then certainly a text/line-based protocol would make sense (possibly in addition to a simple RPC one, if desired). The thing I am worried about is that several corner cases are currently not defined in the proposed protocol and it introduces asynchronous events, which is actually harder to get right in shell scripts than the human protocol.
I generally agree, and to be honest the most useful API for scripts is probably a "qemu-control [OPTIONS] CMD [ARGS...]" _program_ which they can run, along the same lines as apachectl for Apache, smbcontrol for Samba, ntpdc for NTPd, etc.
Such a program can be third party or not, once the monitor behaviour is reliable for a program to use.
-- Jamie
Quoted 1 line · Jamie Lokier
Aren't most QEMU management programs (other than libvirt) scripts?
Probably because we didn't provide an rpc interface before. Chichen and egg problem here...
Quoted 3 lines
Looks to me like "works with libvirt and other management programs" implies that you can use it from scripts, because many management programs are, in fact, scripts.
Having the possibility of using it from scripts would be nice but I wouldn't make it a requirement.
With an rpc interface and the right scripting language you can still use the rpc interface from a very very short script.
Quoted 10 lines · Stefano Stabellini
Quoted 4 lines
Looks to me like "works with libvirt and other management programs" implies that you can use it from scripts, because many management programs are, in fact, scripts.
Having the possibility of using it from scripts would be nice but I wouldn't make it a requirement.
With an rpc interface and the right scripting language you can still use the rpc interface from a very very short script.
I'd go further and make shell script support a non-goal. Maybe the majority of qemu control programs are scripts, but only a tiny minority of VMs are controlled by shell scripts. It is nearly impossible to write a correct nontrivial program in bash; and scripts will be slow due to the need to constantly fork/exec/connect.
Serious management programs will be written in real languages, and that's what we should optimize for.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 12 lines · Avi Kivity
Quoted 10 lines · Stefano Stabellini
Quoted 4 lines
Looks to me like "works with libvirt and other management programs" implies that you can use it from scripts, because many management programs are, in fact, scripts.
Having the possibility of using it from scripts would be nice but I wouldn't make it a requirement.
With an rpc interface and the right scripting language you can still use the rpc interface from a very very short script.
Quoted 3 lines
I'd go further and make shell script support a non-goal. Maybe the majority of qemu control programs are scripts, but only a tiny minority of VMs are controlled by shell scripts. It is nearly impossible to write a correct nontrivial program in bash; and scripts will be slow due to the need to constantly fork/exec/connect.
Serious management programs will be written in real languages, and that's what we should optimize for.
I'm still inclined to stick with the text-based protocol. libvirt likes it. If Serious Management Programs want an RPC interface they can use libvirt.
An RPC is overkill unless you also introduce things like life cycle management. Otherwise, you could never realistically interact with QEMU without a intermediary that handled life cycle management (e.g. libvirt).
Regards,
Anthony Liguori
Quoted 5 lines · Anthony Liguori
Quoted 3 lines
I'd go further and make shell script support a non-goal. Maybe the majority of qemu control programs are scripts, but only a tiny minority of VMs are controlled by shell scripts. It is nearly impossible to write a correct nontrivial program in bash; and scripts will be slow due to the need to constantly fork/exec/connect.
Serious management programs will be written in real languages, and that's what we should optimize for.
I'm still inclined to stick with the text-based protocol. libvirt likes it. If Serious Management Programs want an RPC interface they can use libvirt.
That's really an onerous requirement. What if libvirt lags behind qemu? What if libvirt doesn't want to support some feature?
Quoted 1 line
An RPC is overkill unless you also introduce things like life cycle management. Otherwise, you could never realistically interact with QEMU without a intermediary that handled life cycle management (e.g. libvirt).
QMP is an RPC implemented over a text based protocol, with ad-hoc marshalling.
I agree that you need something to reap dead qemus, but I don't think we can force libvirt on people.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 21 lines · Avi Kivity
Quoted 11 lines · Anthony Liguori
Quoted 8 lines
I'd go further and make shell script support a non-goal. Maybe the majority of qemu control programs are scripts, but only a tiny minority of VMs are controlled by shell scripts. It is nearly impossible to write a correct nontrivial program in bash; and scripts will be slow due to the need to constantly fork/exec/connect.
Serious management programs will be written in real languages, and that's what we should optimize for.
I'm still inclined to stick with the text-based protocol. libvirt likes it. If Serious Management Programs want an RPC interface they can use libvirt.
That's really an onerous requirement. What if libvirt lags behind qemu? What if libvirt doesn't want to support some feature?
Quoted 4 lines
An RPC is overkill unless you also introduce things like life cycle management. Otherwise, you could never realistically interact with QEMU without a intermediary that handled life cycle management (e.g. libvirt).
Quoted 6 lines
QMP is an RPC implemented over a text based protocol, with ad-hoc marshalling.
I agree that you need something to reap dead qemus, but I don't think we can force libvirt on people.
Clearly I agree with Avi.
I am thinking for example that we could use the RPC protocol directly from Xend and I am sure other people will find it useful too.
Quoted 5 lines · Stefano Stabellini
Clearly I agree with Avi.
I am thinking for example that we could use the RPC protocol directly from Xend and I am sure other people will find it useful too.
But you can also use QMP from Xend. In fact, it should be pretty easy to convert the current code in Xend to use QMP.
Regards,
Anthony Liguori
Quoted 9 lines · Anthony Liguori
Quoted 5 lines · Stefano Stabellini
Clearly I agree with Avi.
I am thinking for example that we could use the RPC protocol directly from Xend and I am sure other people will find it useful too.But you can also use QMP from Xend. In fact, it should be pretty easy to convert the current code in Xend to use QMP.
AFAIK, the current XenD code doesn't talk to the QEMU monitor at all, instead having a add-on to QEMU code which pulls info from XenStored. That doesn't alter your point though, it would be pretty easy to make XenD talk to the proposed QMP monitor if desired.
Regards,
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Quoted 11 lines · Daniel P. Berrange
Quoted 9 lines · Anthony Liguori
Quoted 7 lines · Stefano Stabellini
Clearly I agree with Avi.
I am thinking for example that we could use the RPC protocol directlyfrom Xend and I am sure other people will find it useful too.
Quoted 9 lines
Quoted 3 lines
But you can also use QMP from Xend. In fact, it should be pretty easy to convert the current code in Xend to use QMP.
AFAIK, the current XenD code doesn't talk to the QEMU monitor at all, instead having a add-on to QEMU code which pulls info from XenStored. That doesn't alter your point though, it would be pretty easy to make XenD talk to the proposed QMP monitor if desired.
All it takes is implementing QMP, and an emitter/parser for each qemu command. On the other hand, things like xml-rpc require around one line of code in Python per command.
Let's turn this around. IIRC libvirt uses an RPC interface for its clients. How would you estimate the effort to port this interface to QMP, and adapt all its clients?
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 22 lines · Avi Kivity
Quoted 11 lines · Daniel P. Berrange
Quoted 9 lines · Anthony Liguori
Quoted 7 lines · Stefano Stabellini
Clearly I agree with Avi.
I am thinking for example that we could use the RPC protocol directlyfrom Xend and I am sure other people will find it useful too.
Quoted 9 lines
Quoted 3 lines
But you can also use QMP from Xend. In fact, it should be pretty easy to convert the current code in Xend to use QMP.
AFAIK, the current XenD code doesn't talk to the QEMU monitor at all, instead having a add-on to QEMU code which pulls info from XenStored. That doesn't alter your point though, it would be pretty easy to make XenD talk to the proposed QMP monitor if desired.
Quoted 7 lines
All it takes is implementing QMP, and an emitter/parser for each qemu command. On the other hand, things like xml-rpc require around one line of code in Python per command.
Let's turn this around. IIRC libvirt uses an RPC interface for its clients. How would you estimate the effort to port this interface to QMP, and adapt all its clients?
The libvirt RPC interface is a binary message based protocol, using XDR to encode arguments/return values, and emit signals. The libvirt for handling this is quite complex because it has to deal with TLS, and SASL for data encryption, as well as alllowing overlapping RPC requests on a single TCP connection ( to allow multi-threaded clients to have parallelized method calls without opening multiple connections). So it would be a fairly significant amount of work, in comparison to the current QMP proposal.
The existing libvirt code to talk to QEMU's monitor is actually rather simple by comparison to our native RPC code. The problems we have historically had with the QEMU monitor were, ill defined data printed when an error occurs, the changing of monitor commands between QEMU releases in incompatible ways, the lack of quoting for arguments and no support for async events.
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Quoted 6 lines · Daniel P. Berrange
Quoted 5 lines
Let's turn this around. IIRC libvirt uses an RPC interface for its clients. How would you estimate the effort to port this interface to QMP, and adapt all its clients?
Quoted 9 lines
The libvirt RPC interface is a binary message based protocol, using XDR to encode arguments/return values, and emit signals. The libvirt for handling this is quite complex because it has to deal with TLS, and SASL for data encryption, as well as alllowing overlapping RPC requests on a single TCP connection ( to allow multi-threaded clients to have parallelized method calls without opening multiple connections). So it would be a fairly significant amount of work, in comparison to the current QMP proposal.
You're describing an RPC with asynchronous commands and notifications, with support for authentication and authorization. While qemu doesn't need auth*, it does need the other features, so RPC looks like a good fit.
Quoted 7 lines
The existing libvirt code to talk to QEMU's monitor is actually rather simple by comparison to our native RPC code. The problems we have historically had with the QEMU monitor were, ill defined data printed when an error occurs, the changing of monitor commands between QEMU releases in incompatible ways, the lack of quoting for arguments and no support for async events.
No doubt RPC is complicated if you implement it yourself, but surely you used a library? Actually using the RPC is simpler than a line protocol.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 29 lines · Avi Kivity
Quoted 6 lines · Daniel P. Berrange
Quoted 5 lines
Let's turn this around. IIRC libvirt uses an RPC interface for its clients. How would you estimate the effort to port this interface to QMP, and adapt all its clients?
Quoted 9 lines
The libvirt RPC interface is a binary message based protocol, using XDR to encode arguments/return values, and emit signals. The libvirt for handling this is quite complex because it has to deal with TLS, and SASL for data encryption, as well as alllowing overlapping RPC requests on a single TCP connection ( to allow multi-threaded clients to have parallelized method calls without opening multiple connections). So it would be a fairly significant amount of work, in comparison to the current QMP proposal.
You're describing an RPC with asynchronous commands and notifications, with support for authentication and authorization. While qemu doesn't need auth*, it does need the other features, so RPC looks like a good fit.
Quoted 7 lines
The existing libvirt code to talk to QEMU's monitor is actually rather simple by comparison to our native RPC code. The problems we have historically had with the QEMU monitor were, ill defined data printed when an error occurs, the changing of monitor commands between QEMU releases in incompatible ways, the lack of quoting for arguments and no support for async events.
Quoted 2 lines
No doubt RPC is complicated if you implement it yourself, but surely you used a library? Actually using the RPC is simpler than a line protocol.
We only used a library for the data serialization/de-serialization. It would have been nice to use libdbus.so in peer-to-peer (non-bus) mode, but sadly it still lacks any useful form of encryption, or authetnication for TCP, and the windows port is still an incomplete hacked up fork of the code.
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Quoted 1 line · Daniel P. Berrange
We only used a library for the data serialization/de-serialization.
that's what has been proposed here so far, xmlrpc and jsonrpc are just serialization/de-serialization libraries of a "query". dbus is a bit more hence why it got some shortcoming related to qemu related to tcp auth and stuff.
the question is why did you not do your own data seraliazation/deserialization library then ? and then why do your previous answer doesn't apply to qemu too ?
Signature · 1 line
Vincent
Quoted 5 lines · Avi Kivity
Quoted 1 line
An RPC is overkill unless you also introduce things like life cycle management. Otherwise, you could never realistically interact with QEMU without a intermediary that handled life cycle management (e.g. libvirt).
QMP is an RPC implemented over a text based protocol, with ad-hoc marshalling.
I agree that you need something to reap dead qemus, but I don't think we can force libvirt on people.
QMP is an incremental improvement to the current monitor which people are consuming today. It will have a pretty immediate and high rate of return. libvirt can make use of it with probably very little change and it will immediately provide them with new features (async notifications) and better information about commands (error return values).
I have a really hard time not thinking this is the right thing to do because for relatively little churn we'll get a large return. If you just exposed the monitor as an RPC, I don't think you really get anything more useful than QMP and you force a lot more change in libvirt which hurt the adoption of this protocol.
This is exactly what happened with Xend's transition from SExpr/HTTP (custom RPC) -> XML-RPC -> XenAPI. libvirt still uses SExpr/HTTP because the cost of switching to the new way of doing things didn't warrant the return on investment.
s/libvirt/any other management tool that works today/g
Regards,
Anthony Liguori
Quoted 9 lines · Anthony Liguori
Quoted 5 lines · Avi Kivity
Quoted 1 line
An RPC is overkill unless you also introduce things like life cycle management. Otherwise, you could never realistically interact with QEMU without a intermediary that handled life cycle management (e.g. libvirt).
QMP is an RPC implemented over a text based protocol, with ad-hoc marshalling.
I agree that you need something to reap dead qemus, but I don't think we can force libvirt on people.
QMP is an incremental improvement to the current monitor which people are consuming today. It will have a pretty immediate and high rate of return. libvirt can make use of it with probably very little change and it will immediately provide them with new features (async notifications) and better information about commands (error return values).
The incremental cost of porting a monitor command, documenting its format, improving QMP infrastructure where necessary, and implementing that command across all clients is much greater than RPC. For a new client implemented in a high level language, using QMP will be substantially more difficult than using RPC, and the probability of getting things right will be much greater.
Targeting libvirt alone is a mistake. Of course libvirt is a very important user of the qemu monitor, but not doing the right thing because libvirt has some prior art is shortsighted IMO. RPC offers the best choice for growth, in adding new commands, and in supporting more clients.
Quoted 1 line
I have a really hard time not thinking this is the right thing to do because for relatively little churn we'll get a large return. If you just exposed the monitor as an RPC, I don't think you really get anything more useful than QMP and you force a lot more change in libvirt which hurt the adoption of this protocol.
Adopting an RPC should be easier than adopting QMP, since all you have to do is compile the IDL. Again I point to the incremental cost of adding a command.
Quoted 1 line
This is exactly what happened with Xend's transition from SExpr/HTTP (custom RPC) -> XML-RPC -> XenAPI. libvirt still uses SExpr/HTTP because the cost of switching to the new way of doing things didn't warrant the return on investment.
I think this points to libvirt doing something wrong. I suspect it's the implementation language.
Quoted 1 line
s/libvirt/any other management tool that works today/g
Anything written in a high level language (say, Python) will be trivial to port to an RPC. RPC clients in Python can be typed in with one finger in a couple of minutes.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 1 line · Avi Kivity
Adopting an RPC should be easier than adopting QMP, since all you have to do is compile the IDL. Again I point to the incremental cost of adding a command.
Having written clients for XML-RPC, SExpr/HTTP, and the QEMU monitor, in both C and in Python, along with having spent way too much time with various IDL compilers, I really don't agree with you here.
It is relatively easy to parse a line based protocol. Certainly easier than most poorly designed RPCs. For RPCs that are not self-described, you need an IDL compiler to make any sense out of the packets. IDL compilers are a huge problem because you need to build the compiler and run it on the same platform unless you choose from the handful of common compilers. Those compilers/RPCs tend to produce absolutely terrible C code so you need to do all sorts of tricks to avoid excessive warnings.
For self-describing RPCs (like XML-RPC), you can avoid IDL which is good. You still end up with pretty funky C code though and the type system almost always comes back to haunt you.
Quoted 7 lines
Quoted 1 line
This is exactly what happened with Xend's transition from SExpr/HTTP (custom RPC) -> XML-RPC -> XenAPI. libvirt still uses SExpr/HTTP because the cost of switching to the new way of doing things didn't warrant the return on investment.
I think this points to libvirt doing something wrong. I suspect it's the implementation language.
Quoted 1 line
s/libvirt/any other management tool that works today/g
Anything written in a high level language (say, Python) will be trivial to port to an RPC. RPC clients in Python can be typed in with one finger in a couple of minutes.
Reality is less rosy. Take a look at Xend's XML-RPC server/client. Python doesn't provide anything but the most dumb transports. Implementing custom transports requires poking internal bits of the class library. It gets ugly really quickly.
Also, typing really hurts with dynamic languages because the type semantics never match up well which causes unexpected failures. A classic example with XML-RPC is None. There is no standard way to transport None in XML-RPC although there are common extensions to support it. Of course, not all clients support it and there isn't really a standard way to negotiate it. It's really a mess.
At this point, we have a close-to-mergable set of patches that provide real value. I don't see a reason not to merge them once they've been resubmitted.
I don't believe an RPC is going to be a dramatic simplification. It could be if there were a high quality RPC out there that had well maintained client/server libraries. I don't believe such a thing exists though. If you want to keep arguing this, let's focus on actual RPC implementations. I think we've agreed that JSON-RPC isn't going to work because of the lack of client/server library availability. XML-RPC falls into the same boat for C as far as I'm concerned. If there are any other ones you want to consider, let's look at them so that we can make progress in this discussion.
Regards,
Anthony Liguori
Quoted 4 lines · Anthony Liguori
Quoted 1 line · Avi Kivity
Adopting an RPC should be easier than adopting QMP, since all you have to do is compile the IDL. Again I point to the incremental cost of adding a command.
Having written clients for XML-RPC, SExpr/HTTP, and the QEMU monitor, in both C and in Python, along with having spent way too much time with various IDL compilers, I really don't agree with you here.
IDL does suck, but dynamic languages generally don't need it.
Quoted 1 line
It is relatively easy to parse a line based protocol. Certainly easier than most poorly designed RPCs.
The idea is that the parsers have already been written. And I doubt that our RPC masquerading as a line based protocol wouldn't earn the "poorly designed" epithet after a few years of cruft have accumulated.
In Python, adding a new RPC procedure call amounts to typing in the procedure name and its arguments. It really cannot be any simpler than that. On the other hand QMP calls for a hand written emitter and parser for each command.
Quoted 1 line
For RPCs that are not self-described, you need an IDL compiler to make any sense out of the packets. IDL compilers are a huge problem because you need to build the compiler and run it on the same platform unless you choose from the handful of common compilers. Those compilers/RPCs tend to produce absolutely terrible C code so you need to do all sorts of tricks to avoid excessive warnings.
Writing a lot of code is worse than crap code someone else writes.
Quoted 1 line
For self-describing RPCs (like XML-RPC), you can avoid IDL which is good. You still end up with pretty funky C code though and the type system almost always comes back to haunt you.
Don't read that C code.
I doubt we'll have funky type problems.
Quoted 4 lines
Quoted 1 line
Anything written in a high level language (say, Python) will be trivial to port to an RPC. RPC clients in Python can be typed in with one finger in a couple of minutes.
Reality is less rosy. Take a look at Xend's XML-RPC server/client. Python doesn't provide anything but the most dumb transports. Implementing custom transports requires poking internal bits of the class library. It gets ugly really quickly.
I've seen this, and while it fell below by expectations, it's something done once rather than per procedure.
Quoted 1 line
Also, typing really hurts with dynamic languages because the type semantics never match up well which causes unexpected failures. A classic example with XML-RPC is None. There is no standard way to transport None in XML-RPC although there are common extensions to support it. Of course, not all clients support it and there isn't really a standard way to negotiate it. It's really a mess.
With QMP we don't have a standard way to transport anything (the None problem can be avoided by not allowing it at all).
Quoted 1 line
At this point, we have a close-to-mergable set of patches that provide real value. I don't see a reason not to merge them once they've been resubmitted.
Merging is a meaningless milestone since the protocol will not be enabled. Since we'll miss 0.11 in reality we have 7-8 months before the protocol can be used in production. IMO you're optimizing for the short term, disregarding long-term maintenance, and disregarding ease of implementation for users of the qemu monitor.
Quoted 1 line
I don't believe an RPC is going to be a dramatic simplification. It could be if there were a high quality RPC out there that had well maintained client/server libraries. I don't believe such a thing exists though. If you want to keep arguing this, let's focus on actual RPC implementations. I think we've agreed that JSON-RPC isn't going to work because of the lack of client/server library availability. XML-RPC falls into the same boat for C as far as I'm concerned. If there are any other ones you want to consider, let's look at them so that we can make progress in this discussion.
We are not in agreement on jsonrpc (we could easily use a json library and implement the rpc ourselves). Even though I dislike xml-rpc, I'd prefer it to a custom protocol.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 1 line · Avi Kivity
Merging is a meaningless milestone since the protocol will not be enabled. Since we'll miss 0.11 in reality we have 7-8 months before the protocol can be used in production. IMO you're optimizing for the short term, disregarding long-term maintenance, and disregarding ease of implementation for users of the qemu monitor.
QMP is:
__skip__: [ \t]+
symbol: [A-Za-z_\-]+
decimalinteger: [1-9][0-9]+
hexinteger: 0x[0-9]+
float: [0-9]+\.[0-9]+
string: \"([^\"\\]|(\\[0-9][0-9][0-9]))*\" | symbol
number: float | decimalinteger | hexinteger
value: string | number
arg_list: value arg_list?
command: symbol arg_list? '\n'
response_status: ('+' | '-') number arg_list? '\n'response_data: '=' arg_list '\n'async_msg: '*' arg_list '\n'
response: async_msg | response_data* response_status
Client sends 'command's, server responses with 'response'. Responses of response type '*' are server initiated messages. Otherwise, a server always responses to a command with a response type of '+' or '-'. A client sits in a 'response' parsing loop. Each sent 'command' is queued in a FIFO and responses are associated with the commands based on the FIFO order. 'async_msg's do not have associated commands. All strings are UTF-8. Human readable messages depend on the locale used by QEMU.
N.B. there are some subtle changes to Luiz's original proposal to make QMP more well-formed. In particular, more things have to be encoded as strings than previously.
This is a very well formed RPC now. It can be trivially implemented in something like Python. We can add higher level types. For instance, we could:
-value: string | number
+value: (string | number | list | dictionary) +comma_arg_list: value (',' comma_arg_list)? +list: '[' comma_arg_list? ']'
+
+dict_list: string ':' value (',' dict_list)? +dictionary: '{' dict_list? '}'
And then we basically have JSON. The changes on top of Luiz's current patchset would be introducing an error code and adding more structure to monitor_printf() which I asked for anyway.
Regards,
Anthony Liguori
Quoted 3 lines · Anthony Liguori
Quoted 1 line · Avi Kivity
Merging is a meaningless milestone since the protocol will not be enabled. Since we'll miss 0.11 in reality we have 7-8 months before the protocol can be used in production. IMO you're optimizing for the short term, disregarding long-term maintenance, and disregarding ease of implementation for users of the qemu monitor.
Here's an updated grammar.
One benefit of using our own grammar over just JSON-RPC that is line based is that lines form discrete PDUs. If you wanted to implement a JSON-RPC parser, you would have to be able to either invent your own PDU transmission mechanism or rely on the JSON parser to determine message boundaries. Since the later is likely to be a recursive decent parser, this is difficult to work into an asynchronous client library. The same is true for XML-RPC. With XML-RPC, you rely on HTTP's Content-Length header to determine boundaries.
By having a well defined PDU, you can hand lines to the parser and be sure that the recursion will complete without requiring more data.
Regards,
Anthony Liguori
# QMP 1.0
# v1 -> v2 # o allow integer in symbol names (C-style) # o allow negative integer/floats # o no newlines in strings # o one value per response_data line
__skip__: [ \t]+
symbol: [A-Za-z_\-][A-Za-z_\-0-9]*
decimalinteger: (-)?[1-9][0-9]+
hexinteger: 0x[0-9]+
float: (-)?[0-9]+\.[0-9]+
string: \"([^\"\\\n]|(\\[0-9][0-9][0-9]))*\" | symbol
number: float | decimalinteger | hexinteger
comma_arg_list: value (',' comma_arg_list)?
list: '[' comma_arg_list? ']'
dict_list: string ':' value (',' dict_list)?
dictionary: '{' dict_list? '}'
value: (string | number | list | dictionary)
arg_list: value arg_list?
command: symbol arg_list? '\n'
response_status: ('+' | '-') number string? '\n'response_data: '=' value '\n'async_msg: '*' arg_list '\n'
response: async_msg | response_data* response_status
Quoted 58 lines · Anthony Liguori
Quoted 9 lines · Anthony Liguori
Quoted 6 lines · Avi Kivity
Merging is a meaningless milestone since the protocol will not be enabled. Since we'll miss 0.11 in reality we have 7-8 months before the protocol can be used in production. IMO you're optimizing for the short term, disregarding long-term maintenance, and disregarding ease of implementation for users of the qemu monitor.
Here's an updated grammar.
One benefit of using our own grammar over just JSON-RPC that is line based is that lines form discrete PDUs. If you wanted to implement a JSON-RPC parser, you would have to be able to either invent your own PDU transmission mechanism or rely on the JSON parser to determine message boundaries. Since the later is likely to be a recursive decent parser, this is difficult to work into an asynchronous client library. The same is true for XML-RPC. With XML-RPC, you rely on HTTP's Content-Length header to determine boundaries.
By having a well defined PDU, you can hand lines to the parser and be sure that the recursion will complete without requiring more data.
Regards,
Anthony Liguori
# QMP 1.0
# v1 -> v2 # o allow integer in symbol names (C-style) # o allow negative integer/floats # o no newlines in strings # o one value per response_data line
__skip__: [ \t]+symbol: [A-Za-z_\-][A-Za-z_\-0-9]*
decimalinteger: (-)?[1-9][0-9]+
hexinteger: 0x[0-9]+
float: (-)?[0-9]+\.[0-9]+string: \"([^\"\\\n]|(\\[0-9][0-9][0-9]))*\" | symbol
number: float | decimalinteger | hexinteger
comma_arg_list: value (','comma_arg_list)?
list: '['comma_arg_list? ']'
dict_list: string ':' value (','dict_list)?
dictionary: '{'dict_list? '}'value: (string | number | list | dictionary)
arg_list: valuearg_list?
Quoted 9 lines
command: symbol
arg_list? '\n'
response_status: ('+' | '-') number string? '\n'response_data: '=' value '\n'async_msg: '*'arg_list'\n'response:
async_msg| response_data*response_status
Thanks for writing this down, I really appreciate it.
It answers some of the issues I had with the original protocol proposal (interruption of multi-line responses by asynchronous events). What bothers me though is that it's not LL(1) grammar (not a big problem, but writing LL parsers by hand is much easier) and not all floats can be represented (eg. NaN). String is underdefined - does the syntax represent bytes or characters? If characters then the \nnn syntax is insufficient. There's no syntax for binary data which may eventually be required (FDT machine description), but that can be relatively easily added with something like
blob := 'B' base64-string
base64-string := \" ([A-Za-z0-9+/]*) \"
and adjusting "value".
Note that IMAP uses special literal symbols to send binary data and the data transfer must be negotiated (unless LITERAL+ extension is used). It looks like this
S: ... RFC822.BODY {<size>}
C: + Ok, send the data
S: <binary data of the announced size>
It makes the protocol much harder to implement and almost nobody does it right (including major servers), so I would recommend against anything like that. In the ideal case QEMU will never need large enough blobs to warrant it.
I'm still not convinced that reinventing a text-based protocol is way to go, but I welcome the effort for formal specification of what you had in mind.
Best regards,
Filip Navara
Quoted 8 lines · Filip Navara
On Fri, Jun 26, 2009 at 5:02 PM, Anthony Liguori<[email protected]> wrote:
It answers some of the issues I had with the original protocol proposal (interruption of multi-line responses by asynchronous events). What bothers me though is that it's not LL(1) grammar (not a big problem, but writing LL parsers by hand is much easier)
Perhaps you mean LR(1) (i.e. something that can be understood by yacc). When writing parsers by hand, I tend to prefer recursive decent which easily handles LL(k).
Fortunately, this grammar is very simple and can be reasonable converted to LR(1).
Quoted 2 lines
and not
all floats can be represented (eg. NaN).
I'm not sure that's so important. JSON doesn't support NaN.
Quoted 3 lines
String is underdefined - does
the syntax represent bytes or characters? If characters then the \nnn syntax is insufficient.
Strings are UTF-8. Based on the definition of string, the only thing needing escaping is ", \\, and \n. Anything else is just cosmetic. My intention was for \nnn to be a single octal byte as an escape. You could certainly use multiple escape sequences to encoding a large UTF character.
Quoted 4 lines
There's no syntax for binary data which may
eventually be required (FDT machine description), but that can be relatively easily added with something like
I don't think binary data is a requirement. An FDT should be transmitted as a tree, not as a binary blob. You could also transmit binary as a list of bytes though.
Regards,
Anthony Liguori
Quoted 7 lines · Anthony Liguori
Quoted 4 lines · Filip Navara
On Fri, Jun 26, 2009 at 5:02 PM, Anthony Liguori<[email protected]> wrote:
Quoted 8 lines
Quoted 4 lines
It answers some of the issues I had with the original protocol proposal (interruption of multi-line responses by asynchronous events). What bothers me though is that it's not LL(1) grammar (not a big problem, but writing LL parsers by hand is much easier)
Perhaps you mean LR(1) (i.e. something that can be understood by yacc). When writing parsers by hand, I tend to prefer recursive decent which easily handles LL(k).
I meant LL(1). Nevertheless I realized that it's easy to write a lexer and LL(1) parser given this grammar, so ignore this objection. I prefer to write my parsers by hand instead of using yacc or other tools as long as the grammar is simple enough.
Quoted 7 lines
Fortunately, this grammar is very simple and can be reasonable converted to LR(1).
Quoted 2 lines
and not
all floats can be represented (eg. NaN).I'm not sure that's so important. JSON doesn't support NaN.
It may not be important now, but I consider it important to get the syntax right the first time. Adding and extending commands to existing implementation is easy (or easier at least) and can be done in backward-compatible way if the syntax of the protocol doesn't change between versions. What JSON does or doesn't support is irrelevant here, most platforms QEMU runs on use IEEE 754 floats and if you want to express them in the syntax then express all of their features - Inf, NaN and exponents (maybe I forgot something). The benefit is that you can use printf functions to output them if the syntax matches what C uses.
Quoted 8 lines
Quoted 3 lines
String is underdefined - does
the syntax represent bytes or characters? If characters then the \nnn syntax is insufficient.Strings are UTF-8. Based on the definition of string, the only thing needing escaping is ", \\, and \n. Anything else is just cosmetic. My intention was for \nnn to be a single octal byte as an escape. You could certainly use multiple escape sequences to encoding a large UTF character.
I guess you mean that the "string" in grammar is sequence of bytes. These bytes should be interpreted as UTF-8 characters then.
Note that this is very important to specify. For example the GREEK CAPITAL LETTER PI is unicode code point 928. If the "string" syntax was meant to represent characters I would expect "\928" to refer to single character string with PI being the character, on the other hand in byte-based UTF-8 string you'd write "\206\160".
Quoted 9 lines
Quoted 5 lines
There's no syntax for binary data which may
eventually be required (FDT machine description), but that can be relatively easily added with something likeI don't think binary data is a requirement. An FDT should be transmitted as a tree, not as a binary blob. You could also transmit binary as a list of bytes though.
You can transmit it as list of bytes and it's woefully inefficient. I gave the FDT as an example, but I believe that binary data may be needed in future for one purpose or another. The monitor already provides a way to dump guest memory and I see no reason to not transfer it using some "binary" encoding.
Quoted 5 lines
Regards,
Anthony Liguori
Regards,
Filip Navara
Quoted 11 lines · Filip Navara
Quoted 4 lines
I don't think binary data is a requirement. An FDT should be transmitted as a tree, not as a binary blob. You could also transmit binary as a list of bytes though.
You can transmit it as list of bytes and it's woefully inefficient. I gave the FDT as an example, but I believe that binary data may be needed in future for one purpose or another. The monitor already provides a way to dump guest memory and I see no reason to not transfer it using some "binary" encoding.
Another candidate is screendump. There's no reason to go through a file. We could pass a pipe as an fd and request a screendump to that fd, but that's rather roundabout.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 6 lines · Avi Kivity
Quoted 5 lines · Filip Navara
Quoted 4 lines
I don't think binary data is a requirement. An FDT should be transmitted as a tree, not as a binary blob. You could also transmit binary as a list of bytes though.
Quoted 8 lines
Quoted 6 lines
You can transmit it as list of bytes and it's woefully inefficient. I gave the FDT as an example, but I believe that binary data may be needed in future for one purpose or another. The monitor already provides a way to dump guest memory and I see no reason to not transfer it using some "binary" encoding.
Another candidate is screendump. There's no reason to go through a file. We could pass a pipe as an fd and request a screendump to that fd, but that's rather roundabout.
Instead of inventing binary syntaxes, maybe we could use dictionary syntax to denote a mime encoded value?
So something like:
{"Content-Type": "image/ppm", "Content-Transfer-Encoding": "base64", "Content": "AB234SDFSDf=="}
A special brace format could be used if we wanted to make sure it was independently parsable:
<"Content-Type": "image/ppm", "Content-Transfer-Encoding": "base64", "Content": "AB234SDFSDf==">
But I suspect dicts are good enough.
Regards,
Anthony Liguori
Quoted 7 lines · Anthony Liguori
Quoted 1 line
Another candidate is screendump. There's no reason to go through a file. We could pass a pipe as an fd and request a screendump to that fd, but that's rather roundabout.
Instead of inventing binary syntaxes, maybe we could use dictionary syntax to denote a mime encoded value?
So something like:
Quoted 7 lines
{"Content-Type": "image/ppm", "Content-Transfer-Encoding": "base64", "Content": "AB234SDFSDf=="}
A special brace format could be used if we wanted to make sure it was independently parsable:
<"Content-Type": "image/ppm", "Content-Transfer-Encoding": "base64", "Content": "AB234SDFSDf==">
But I suspect dicts are good enough.
Yes.
(I see a future thread about how we negotiate content types etc.)
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 8 lines · Anthony Liguori
Instead of inventing binary syntaxes, maybe we could use dictionary syntax to denote a mime encoded value?
So something like:
{"Content-Type": "image/ppm", "Content-Transfer-Encoding": "base64", "Content": "AB234SDFSDf=="}
Quoted 8 lines
A special brace format could be used if we wanted to make sure it was independently parsable:
<"Content-Type": "image/ppm", "Content-Transfer-Encoding": "base64", "Content": "AB234SDFSDf==">
But I suspect dicts are good enough.
I think it would be a good idea to allow binary as Content-Transfer-Encoding, since base64 is very inefficient for transferring large amount of data.
For example we may want to save the state of the guest, including the whole memory, and transfer it over the network without saving it to file first.
Quoted 7 lines · Stefano Stabellini
I think it would be a good idea to allow binary as Content-Transfer-Encoding, since base64 is very inefficient for transferring large amount of data.
For example we may want to save the state of the guest, including the whole memory, and transfer it over the network without saving it to file first.
The migrate command does that. Give it an fd and migrate away.
Mixing binary in the data format seems a bad idea to me, especially if we stick to json as our encoding.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 9 lines · Avi Kivity
Quoted 7 lines · Stefano Stabellini
I think it would be a good idea to allow binary as Content-Transfer-Encoding, since base64 is very inefficient for transferring large amount of data.
For example we may want to save the state of the guest, including the whole memory, and transfer it over the network without saving it to file first.
Quoted 5 lines
The migrate command does that. Give it an fd and migrate away.
Mixing binary in the data format seems a bad idea to me, especially if we stick to json as our encoding.
JSON compatibility is more important to me than binary data, but as a reply to the discussion about mime encodings, I was just pointing out that mime does support binary encodings:
http://tools.ietf.org/html/rfc1652
http://tools.ietf.org/html/rfc3030
and if it doesn't prove to be too hard we could start thinking about supporting them.
Quoted 5 lines · Anthony Liguori
Quoted 2 lines · Filip Navara
and not
all floats can be represented (eg. NaN).I'm not sure that's so important. JSON doesn't support NaN.
BTW, JSON doesn't even pretend to support floats. What it supports is a notation to write decimal fractions, in many languages that maps to the DECIMAL type (number with sign, mantissa and exponent in base-10) and not float. Calling it float is a misnomer.
Best regards,
Filip Navara
Quoted 7 lines · Anthony Liguori
Quoted 3 lines · Anthony Liguori
Quoted 1 line · Avi Kivity
Merging is a meaningless milestone since the protocol will not be enabled. Since we'll miss 0.11 in reality we have 7-8 months before the protocol can be used in production. IMO you're optimizing for the short term, disregarding long-term maintenance, and disregarding ease of implementation for users of the qemu monitor.
Here's an updated grammar.
One benefit of using our own grammar over just JSON-RPC that is line based is that lines form discrete PDUs. If you wanted to implement a JSON-RPC parser, you would have to be able to either invent your own PDU transmission mechanism or rely on the JSON parser to determine message boundaries. Since the later is likely to be a recursive decent parser, this is difficult to work into an asynchronous client library. The same is true for XML-RPC. With XML-RPC, you rely on HTTP's Content-Length header to determine boundaries.
You could also abort the parse if you reach EOF, and start it from the beginning when you get more data. It's reasonably efficient in a local protocol, and when you expect small messages.
I'm fine with a newline delimiter though.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 6 lines · Anthony Liguori
# v1 -> v2 # o allow integer in symbol names (C-style) # o allow negative integer/floats # o no newlines in strings # o one value per response_data line
Quoted 7 lines
__skip__: [ \t]+symbol: [A-Za-z_\-][A-Za-z_\-0-9]*
decimalinteger: (-)?[1-9][0-9]+
Traditionally, 0 is recognized as an integer.
Quoted 2 lines
hexinteger: 0x[0-9]+
[a-fA-f]
Quoted 2 lines
float: (-)?[0-9]+\.[0-9]+
5, 3., .10, 1e9 are all floats.
Quoted 2 lines
string: \"([^\"\\\n]|(\\[0-9][0-9][0-9]))*\" | symbol
I think json recognises 'blah' but not symbol. Also it may recognize other escapes.
It's important to allow clients to use json emitters so they can construct a dict and throw it at qemu, similarly to parse qemu results.
Quoted 4 lines
arg_list: valuearg_list?command: symbol
arg_list? '\n'
Better to use a dictionary for the argument list (more self-documenting, easier expansion).
I'd also adopt jsonrpc's request and id attributes, they allow async commands and remove the need for any specialized parsing.
Quoted 5 lines
response_status: ('+' | '-') number string? '\n'response_data: '=' value '\n'async_msg: '*'arg_list'\n'
Similarly, return a value that contains message type, id, and return value.
I think it's fine to drop jsonrpc as the standard (but retain its features) and just adopt json for encoding.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Avi Kivity
Quoted 10 lines · Anthony Liguori
# v1 -> v2 # o allow integer in symbol names (C-style) # o allow negative integer/floats # o no newlines in strings # o one value per response_data line
__skip__: [ \t]+symbol: [A-Za-z_\-][A-Za-z_\-0-9]*
Quoted 8 lines
Quoted 2 lines
decimalinteger: (-)?[1-9][0-9]+
Traditionally, 0 is recognized as an integer.
Quoted 2 lines
hexinteger: 0x[0-9]+
[a-fA-f]
I have an updated grammer that has both of these fixes.
Quoted 4 lines
Quoted 2 lines
float: (-)?[0-9]+\.[0-9]+
5, 3., .10, 1e9 are all floats.
I was following JSON style floats as opposed to C float. C floats are irregular and difficult to parse with an automatic parser. If we care, it's worth doing correctly though.
Quoted 4 lines
Quoted 2 lines
string: \"([^\"\\\n]|(\\[0-9][0-9][0-9]))*\" | symbol
I think json recognises 'blah' but not symbol. Also it may recognize other escapes.
We can drop symbol. The original intention was to output format similar to what we have today but we're pretty far from that now.
Quoted 8 lines
It's important to allow clients to use json emitters so they can construct a dict and throw it at qemu, similarly to parse qemu results.
Quoted 4 lines
arg_list: valuearg_list?command: symbol
arg_list? '\n'Better to use a dictionary for the argument list (more self-documenting, easier expansion).
Verses a list? I'm not sure a dict fits the current QEMU model very well. How would you propose fitting it into the current monitor infrastructure?
Quoted 1 line
I'd also adopt jsonrpc's request and id attributes, they allow async commands and remove the need for any specialized parsing.
I think command tagging is a very good thing to do.
Quoted 7 lines
Quoted 5 lines
response_status: ('+' | '-') number string? '\n'response_data: '=' value '\n'async_msg: '*'arg_list'\n'Similarly, return a value that contains message type, id, and return value.
I think command outputs are basically return values. What's nice about the current QMP proposal is that we're basically returning tuples.
Quoted 1 line
I think it's fine to drop jsonrpc as the standard (but retain its features) and just adopt json for encoding.
Yeah, I think if we focus on defining a grammar and semantics, if it happens to look like JSON, I don't really care. I just don't want to adopt the baggage of json and particularly json-rpc.
Regards,
Anthony Liguori
Quoted 9 lines · Anthony Liguori
Quoted 7 lines
It's important to allow clients to use json emitters so they can construct a dict and throw it at qemu, similarly to parse qemu results.
Quoted 3 lines
arg_list: valuearg_list?command: symbol
arg_list? '\n'Better to use a dictionary for the argument list (more self-documenting, easier expansion).
Verses a list? I'm not sure a dict fits the current QEMU model very well. How would you propose fitting it into the current monitor infrastructure?
First, I would like to consider the protocol mostly from the point of view of clients, not qemu, and certainly not current qemu code. There is only one qemu, but many clients. Code problems can be solved tomorrow, but interface problems remain forever.
Second, I think it fits quite nicely. Positional arguments are nasty when you have many of them, especially when some are optional. And we already support dictionaries: -drive file=blah,cache=none. We make a dictionary interface, and have both the command line parser and the monitor json thing implement it.
(note: it would be nice to have this interface detect unused arguments so we can flag typos in keywords).
Quoted 3 lines
Quoted 1 line
Similarly, return a value that contains message type, id, and return value.
I think command outputs are basically return values. What's nice about the current QMP proposal is that we're basically returning tuples.
Yes, I meant { 'type': 'respose', 'id': 'foo', 'return': value }. 'type' is the new =, and 'id' is the tag. If multiple return values are needed, value can be a list or dictionary.
Quoted 3 lines
Quoted 1 line
I think it's fine to drop jsonrpc as the standard (but retain its features) and just adopt json for encoding.
Yeah, I think if we focus on defining a grammar and semantics, if it happens to look like JSON, I don't really care. I just don't want to adopt the baggage of json and particularly json-rpc.
If 'looks like' == 'compatible with', I agree.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 1 line · Avi Kivity
First, I would like to consider the protocol mostly from the point of view of clients, not qemu, and certainly not current qemu code. There is only one qemu, but many clients. Code problems can be solved tomorrow, but interface problems remain forever.
I'm trying to consider both. We need something that we can get into a robust state in 6 months. I don't want to undertake a massive effort that will require major changes because history shows that we'll end up with something that doesn't really do the job.
Quoted 1 line
Second, I think it fits quite nicely. Positional arguments are nasty when you have many of them, especially when some are optional. And we already support dictionaries: -drive file=blah,cache=none. We make a dictionary interface, and have both the command line parser and the monitor json thing implement it.
It a fair bit of complexity but I don't think it's too problematic. The current monitor abstraction doesn't have a mechanism to define names of arguments but I think it could be added relatively easily although it wouldn't be so elegant (another parameter with comma deliminated names would do the trick).
Quoted 2 lines
Yes, I meant { 'type': 'respose', 'id': 'foo', 'return': value }. 'type' is the new =, and 'id' is the tag. If multiple return values are needed, value can be a list or dictionary.
I really don't like the syntax you proposed but that's an aesthetic problem. This is an interface consumed by non humans so I don't think it's worth arguing about something that is just as easily parsed by a computer even if it's hideously ugly :-)
So I'll defer to whatever Luiz (or whoever sends patches) decides to do as long as it meets the same requirements.
The main thing missing from the first patchset that this thread has revealed is a formal grammar for the data. This is an important requirement to ensure that it's reasonably parse-able by both ends.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
I think it's fine to drop jsonrpc as the standard (but retain its features) and just adopt json for encoding.
Yeah, I think if we focus on defining a grammar and semantics, if it happens to look like JSON, I don't really care. I just don't want to adopt the baggage of json and particularly json-rpc.
If 'looks like' == 'compatible with', I agree.
You lost me here. JSON interoperability cannot be a requirement. Otherwise, we're going to get stuck dealing with all the warts of JSON. I don't want people to assume they can just use a JSON parser.
You seem to have conflicting requirements. You want JSON compatibility but then you want C-style floats which JSON doesn't support.
I don't mind if this ends up looking like JSON, but I don't want to have to enforce that it's remains JSON compatible in the long term. We should have our own grammar that we can version and that people can use as the basis of a client.
If being pure JSON is important to someone (and I don't think it will be), then someone can write a JSON proxy. You don't need any RPC knowledge to do that so it can be relatively future-proof.
Regards,
Anthony Liguori
Quoted 4 lines · Anthony Liguori
Quoted 1 line · Avi Kivity
First, I would like to consider the protocol mostly from the point of view of clients, not qemu, and certainly not current qemu code. There is only one qemu, but many clients. Code problems can be solved tomorrow, but interface problems remain forever.
I'm trying to consider both. We need something that we can get into a robust state in 6 months. I don't want to undertake a massive effort that will require major changes because history shows that we'll end up with something that doesn't really do the job.
History also shows that band aids rarely work, that trying to reinvent wheels leads to unusual shapes, and that we're pretty bad at predicting future needs.
Quoted 3 lines
Quoted 1 line
Second, I think it fits quite nicely. Positional arguments are nasty when you have many of them, especially when some are optional. And we already support dictionaries: -drive file=blah,cache=none. We make a dictionary interface, and have both the command line parser and the monitor json thing implement it.
It a fair bit of complexity but I don't think it's too problematic. The current monitor abstraction doesn't have a mechanism to define names of arguments but I think it could be added relatively easily although it wouldn't be so elegant (another parameter with comma deliminated names would do the trick).
Instead of 7 callback types you get one callback type with one parameter (an object). We can retain the table driven parsing for the human monitor by, for example, changing
{ "memsave", "lis", do_memory_save,
"addr size file", "save to disk virtual memory dump starting at
'addr' of size 'size'", },
to
{ "memsave", "address:l,size:i,file:s", do_memory_save,
"save to disk virtual memory dump starting at 'address' of size
'size'", },
Quoted 4 lines
Quoted 2 lines
Yes, I meant { 'type': 'respose', 'id': 'foo', 'return': value }. 'type' is the new =, and 'id' is the tag. If multiple return values are needed, value can be a list or dictionary.
I really don't like the syntax you proposed but that's an aesthetic problem. This is an interface consumed by non humans so I don't think it's worth arguing about something that is just as easily parsed by a computer even if it's hideously ugly :-)
I too prefer looking at the original QMP with hand crafted syntax. But even more I prefer not looking at it, and the more we make it machine consumable, the less we have to dig in.
Quoted 1 line
The main thing missing from the first patchset that this thread has revealed is a formal grammar for the data. This is an important requirement to ensure that it's reasonably parse-able by both ends.
That's something else gained from json -- someone else has written the grammar and made sure it works in real life.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
I think it's fine to drop jsonrpc as the standard (but retain its features) and just adopt json for encoding.
Yeah, I think if we focus on defining a grammar and semantics, if it happens to look like JSON, I don't really care. I just don't want to adopt the baggage of json and particularly json-rpc.
If 'looks like' == 'compatible with', I agree.
You lost me here. JSON interoperability cannot be a requirement. Otherwise, we're going to get stuck dealing with all the warts of JSON.
Look at the top half of http://www.json.org/. It's remarkably wart free.
Quoted 1 line
I don't want people to assume they can just use a JSON parser.
Look at the bottom half of http://www.json.org/. It's remarkably parser full. It means that anyone can get up and running writing a cloud in haXe or Fan without studying and implementing the qemu data format.
Quoted 2 lines
You seem to have conflicting requirements. You want JSON compatibility but then you want C-style floats which JSON doesn't support.
According to the json grammar, it supports floats (including exponents), except for NaN and Inf. Personally, I'm happy to limit qemu to finite numbers.
Quoted 1 line
I don't mind if this ends up looking like JSON, but I don't want to have to enforce that it's remains JSON compatible in the long term. We should have our own grammar that we can version and that people can use as the basis of a client.
That really reduces the attractiveness of the whole thing. Writing parsers and emitters should be an optional part of writing a qemu control program, not a required part.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 9 lines · Avi Kivity
Quoted 4 lines
I don't mind if this ends up looking like JSON, but I don't want to have to enforce that it's remains JSON compatible in the long term. We should have our own grammar that we can version and that people can use as the basis of a client.
That really reduces the attractiveness of the whole thing. Writing parsers and emitters should be an optional part of writing a qemu control program, not a required part.
I think that using JSON syntax, hence re-using JSON parsers and emitters, it is a good compromise between supporting a full-blown RPC and implementing our own IMAP like protocol.
This way both parties should be satisfied.
Quoted 3 lines · Avi Kivity
Quoted 1 line
I don't mind if this ends up looking like JSON, but I don't want to have to enforce that it's remains JSON compatible in the long term. We should have our own grammar that we can version and that people can use as the basis of a client.
That really reduces the attractiveness of the whole thing. Writing parsers and emitters should be an optional part of writing a qemu control program, not a required part.
I really disagree but this thread is also killing my soul :-)
Whatever we do for QMP, it will not be enabled for 0.11 so we have 6 months to get it right. In the interest of moving forward and writing patches instead of emails, here's what I'm thinking:
1) Update the QMP patches to support return values for monitor commands
2) Update patches to support structured command output
3) Update the patches to support higher level data types (like lists and dictionaries)
4) For whatever the emission format is, provide a regular grammar to parse that output
I will commit a patch series that meets these goals.
Given a regular grammar, it's extremely easy to convert to outputting JSON, XML, or whatever. We can keep arguing about whether JSON is the right format long term. However, I don't want the useful work of conversing monitor commands to satisify 1-3 to be held up by arguments about the emission format which is largely unrelated to the command conversions.
Regards,
Anthony Liguori
Quoted 5 lines · Anthony Liguori
Quoted 3 lines · Avi Kivity
Quoted 1 line
I don't mind if this ends up looking like JSON, but I don't want to have to enforce that it's remains JSON compatible in the long term. We should have our own grammar that we can version and that people can use as the basis of a client.
That really reduces the attractiveness of the whole thing. Writing parsers and emitters should be an optional part of writing a qemu control program, not a required part.
I really disagree
Writing parsers and emitters should be a required part of writing a qemu control program?
Quoted 9 lines
Whatever we do for QMP, it will not be enabled for 0.11 so we have 6 months to get it right. In the interest of moving forward and writing patches instead of emails, here's what I'm thinking:
1) Update the QMP patches to support return values for monitor commands
2) Update patches to support structured command output
3) Update the patches to support higher level data types (like lists and dictionaries)
4) For whatever the emission format is, provide a regular grammar to parse that outputI will commit a patch series that meets these goals.
We have six months so you'll commit some unreviewed patches now?
This is deeply dissatisfying.
Quoted 1 line
Given a regular grammar, it's extremely easy to convert to outputting JSON, XML, or whatever. We can keep arguing about whether JSON is the right format long term. However, I don't want the useful work of conversing monitor commands to satisify 1-3 to be held up by arguments about the emission format which is largely unrelated to the command conversions.
We can certainly get consensus on some things earlier. I don't see the need to commit without review though.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Quoted 1 line · Avi Kivity
Writing parsers and emitters should be a required part of writing a qemu control program?
That's what libraries are for. The thing that you still haven't addressed with JSON, is that there isn't a huge community of JSON parsers out there. The only json library available in Fedora 10 for C appears to be json-glib which carries a GObject dependency. Practically speaking, if we did use JSON, we would have to provide our own parser library.
Quoted 3 lines
Quoted 1 line
I will commit a patch series that meets these goals.
We have six months so you'll commit some unreviewed patches now?
Where does unreviewed come from? Why does the emission format have to be perfect in order to start refactoring the monitor interfaces? There is a large bit of this work effort that has nothing to do with output format. I don't want to block that work effort.
Regards,
Anthony Liguori
Quoted 11 lines · Anthony Liguori
Quoted 2 lines · Avi Kivity
Writing parsers and emitters should be a required part of writing a qemu control program?
That's what libraries are for. The thing that you still haven't addressed with JSON, is that there isn't a huge community of JSON parsers out there. The only json library available in Fedora 10 for C appears to be json-glib which carries a GObject dependency. Practically speaking, if we did use JSON, we would have to provide our own parser library.
I am just re-posting this portion of a week old email to qemu-devel from Vincent, about C libraries for JSON, because this week he is on vacation:
Quoted 7 lines
it might be missing for C, but lots of language got a maintained and used library nowadays. We're using it in ocaml, and i have to say we haven't been disappointed using it, compared to the number of disappointments with xmlrpc (and mostly xml in general).
That said, I'll make a C library even if qemu doesn't end up using it.
Quoted 4 lines · Anthony Liguori
Quoted 1 line · Avi Kivity
Writing parsers and emitters should be a required part of writing a qemu control program?
That's what libraries are for. The thing that you still haven't addressed with JSON, is that there isn't a huge community of JSON parsers out there. The only json library available in Fedora 10 for C appears to be json-glib which carries a GObject dependency. Practically speaking, if we did use JSON, we would have to provide our own parser library.
There are around 100 parsers listed on json.org. We can make the machine protocol optional and dependant on the existence of a json parser of our choice, and Fedora will have to package that dependency if they want to build the machine protocol.
Clients are free to use a library or write their own parser. If we provided a parser library it would be useless for all but one client. If we support a common format, clients can reuse an existing library.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
I will commit a patch series that meets these goals.
We have six months so you'll commit some unreviewed patches now?
Where does unreviewed come from?
Sorry, I understood "I will commit" as "I will personally write and commit", not as "I will commit Luiz's suitably modified patchset".
Quoted 1 line
Why does the emission format have to be perfect in order to start refactoring the monitor interfaces? There is a large bit of this work effort that has nothing to do with output format. I don't want to block that work effort.
It does not, and modifying the internals to support a serializable object model can be done in parallel to the json flamewar.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 3 lines · Avi Kivity
Quoted 1 line
Why does the emission format have to be perfect in order to start refactoring the monitor interfaces? There is a large bit of this work effort that has nothing to do with output format. I don't want to block that work effort.
It does not, and modifying the internals to support a serializable object model can be done in parallel to the json flamewar.
Great. So Luiz, do you understand and agree with the proposed changes to your series?
Regards,
Anthony Liguori
Quoted 11 lines · On Tue, 30 Jun 2009 08:56:26 -0500 Anthony Liguori <[email protected]>…
Quoted 7 lines · Avi Kivity
Quoted 4 lines
Why does the emission format have to be perfect in order to start refactoring the monitor interfaces? There is a large bit of this work effort that has nothing to do with output format. I don't want to block that work effort.
It does not, and modifying the internals to support a serializable object model can be done in parallel to the json flamewar.
Great. So Luiz, do you understand and agree with the proposed changes to your series?
Yes, only the lists and dictionaries types are not very clear
to me.
Should command handlers return data in those types to the
monitor and thus get printed by it?
Detailed is welcome, anyway.
Quoted 4 lines · Luiz Capitulino
Quoted 3 lines
Great. So Luiz, do you understand and agree with the proposed changes to your series?
Quoted 8 lines
Yes, only the lists and dictionaries types are not very clear
to me.Should command handlers return data in those types to the
monitor and thus get printed by it?Detailed is welcome, anyway.
I would recommend proceeding as follows:
1. Add QValue, QArray, QDictionary
QValue is a polymorphic object that can be a number, a string, an array, or a dictionary. It needs a bunch of accessors like qvalue_to_qarray() to check if a value is an array, and things like qarray_get() or qdict_put() to manipulate them. For the simpler data types, qvalue_to_int64() and qvalue_from_int64() should suffice.
2. Split the human monitor command implementations into protocol adapters and implementation.
The protocol adapter reads the command, parses it according to qemu-monitor.hx, and constructs a QArray of the parameter list and passes it down to the command implementation. The command implementation returns a QValue, which the protocol adapter formats to the human monitor response format.
3. Refine the object model
Instead of passing a QArray for the argument list, pass a QDictionary. For arguments that are themselves dictionaries (file=blah.img,cache=none), pass them as dictionaries instead of strings ({file: blah.img, cache: none}).
When the dust settles down on the encoding selection, we simply make the machine protocol parser generate QValues and pass these to the newly-splitted command implementations. We pass the return value to an emitter of our choice which can convert an arbitrary QValue into a string representation.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Avi Kivity
Quoted 10 lines · Luiz Capitulino
Quoted 3 lines
Great. So Luiz, do you understand and agree with the proposed changes to your series?
Yes, only the lists and dictionaries types are not very clear
to me.Should command handlers return data in those types to the
monitor and thus get printed by it?
Quoted 8 lines
Quoted 2 lines
Detailed is welcome, anyway.
I would recommend proceeding as follows:
1. Add QValue, QArray, QDictionary
QValue is a polymorphic object that can be a number, a string, an array, or a dictionary. It needs a bunch of accessors like
qvalue_to_qarray()to check if a value is an array, and things likeqarray_get()orqdict_put()to manipulate them. For the simpler data types,qvalue_to_int64()andqvalue_from_int64()should suffice.
And add a return value to the monitor functions. I think this is a good first step.
Passing a QArray to each monitor function is one way to go. Another way to go is to use something like libffi to do proper dynamic dispatch. The later approach has the benefit of requiring less code churn and reusing the static type checking.
Quoted 3 lines
2. Split the human monitor command implementations into protocol adapters and implementation.
The protocol adapter reads the command, parses it according to qemu-monitor.hx, and constructs a QArray of the parameter list and passes it down to the command implementation. The command implementation returns a QValue, which the protocol adapter formats to the human monitor response format.
One place to start would be to convert something like:
monitor_printf(mon, "file: f=%s,b=%s,c=%d", ...)
to:
value = qlist_create(qstring_create("file"), qdict_create("f", qstring_create(..), "b", qstring_create(..), "c", qstring_create(..), NULL));
But since this is a very regular pattern, you could also do:
value = qmonlist_create("file",
"f", QVALUE_STRING, ...,
"b", QVALUE_STRING, ...,
"c", QVALUE_INT, ...,
NULL);You could potentially introduce some serious ninja action by doing something like:
value = qvalue_create("[%s, {'f': %s, 'b': %s', 'c': %d}]", ...);
What's cool about this is that you can mark qvalue_create() as having printf formatting so GCC will help you with type checking.
Regards,
Anthony Liguori
Quoted 8 lines · Anthony Liguori
Quoted 3 lines
1. Add QValue, QArray, QDictionary
QValue is a polymorphic object that can be a number, a string, an array, or a dictionary. It needs a bunch of accessors like
qvalue_to_qarray()to check if a value is an array, and things likeqarray_get()orqdict_put()to manipulate them. For the simpler data types,qvalue_to_int64()andqvalue_from_int64()should suffice.And add a return value to the monitor functions. I think this is a good first step.
Passing a QArray to each monitor function is one way to go. Another way to go is to use something like libffi to do proper dynamic dispatch. The later approach has the benefit of requiring less code churn and reusing the static type checking.
If/when we switch to passing dictionaries instead of arrays, this won't work. I also think libffi is a little to exotic for something like the monitor. You could potentially introduce some serious ninja action by doing something like:
Quoted 1 line
value =
qvalue_create("[%s, {'f': %s, 'b': %s', 'c': %d}]", ...);
Seriously nifty...
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 10 lines · Anthony Liguori
Quoted 4 lines
Yes, I meant { 'type': 'respose', 'id': 'foo', 'return': value }. 'type' is the new =, and 'id' is the tag. If multiple return values are needed, value can be a list or dictionary.
I really don't like the syntax you proposed but that's an aesthetic problem. This is an interface consumed by non humans so I don't think it's worth arguing about something that is just as easily parsed by a computer even if it's hideously ugly :-)
Quoted 7 lines
So I'll defer to whatever Luiz (or whoever sends patches) decides to do as long as it meets the same requirements.
The main thing missing from the first patchset that this thread has revealed is a formal grammar for the data. This is an important requirement to ensure that it's reasonably parse-able by both ends.
Even though it is not a fun thing to do I think the interface should be written in a more formal specification language than a spec.txt or a C header file.
We can even come up with our own language to do it as long as it forces us to maintain a robust interface and allow us to write simple scripts to validate the methods we use to implement the interface in our own language of choice.
Quoted 8 lines · Stefano Stabellini
Even though it is not a fun thing to do I think the interface should be written in a more formal specification language than a spec.txt or a C header file.
We can even come up with our own language to do it as long as it forces us to maintain a robust interface and allow us to write simple scripts to validate the methods we use to implement the interface in our own language of choice.
I agree. It's really easy to screw up interfaces. It can be a simple validator that runs on requests and responses, or a full fledged IDL compiler, but we need something. I don't think this should block merging so long as it's added before 0.12.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 1 line · Anthony Liguori
Avi Kivity wrote:
[snip]
Quoted 5 lines
Quoted 1 line
5, 3., .10, 1e9 are all floats.
I was following JSON style floats as opposed to C float. C floats are irregular and difficult to parse with an automatic parser. If we care, it's worth doing correctly though.
I'd very much prefer to do it correctly and express all C floats in the syntax.
Going back to my analogy with IMAP, the IMAP syntax doesn't allow for multipart messages with zero parts. While this doesn't sound like anything useful it's a discrepancy with the MIME standard of multipart messages which allows it. And since the MIME standard allows it, applications started to use it (some MMS operator gateways iirc). The result is that instead of expressing the MIME structure directly in IMAP (as was the original intent) one has to first convert it into something that can be expressed in the IMAP syntax (by adding dummy parts to the message).
Best regards,
Filip Navara
Quoted 8 lines · Anthony Liguori
Quoted 3 lines · Avi Kivity
Adopting an RPC should be easier than adopting QMP, since all you have to do is compile the IDL. Again I point to the incremental cost of adding a command.
Having written clients for XML-RPC, SExpr/HTTP, and the QEMU monitor, in both C and in Python, along with having spent way too much time with various IDL compilers, I really don't agree with you here.
did you write any ad-hoc text base protocol then ?
Quoted 1 line
It is relatively easy to parse a line based protocol.
it's not that hard indeed, but now what's the cost of having to replicate this ad-hoc protocol in the management layer that use different languages ? what about the cost of extensibility in the future (hello IMAP) ?
Quoted 11 lines
Certainly easier
than most poorly designed RPCs. For RPCs that are not self-described, you need an IDL compiler to make any sense out of the packets. IDL compilers are a huge problem because you need to build the compiler and run it on the same platform unless you choose from the handful of common compilers. Those compilers/RPCs tend to produce absolutely terrible C code so you need to do all sorts of tricks to avoid excessive warnings.For self-describing RPCs (like XML-RPC), you can avoid IDL which is good. You still end up with pretty funky C code though and the type system almost always comes back to haunt you.
you can avoid IDL in all case, if you choose. it requires more boilerplate in C obviously, but looking at QMP .. it's adding exactly the same boilerplate.
Quoted 4 lines
Reality is less rosy. Take a look at Xend's XML-RPC server/client. Python doesn't provide anything but the most dumb transports. Implementing custom transports requires poking internal bits of the class library. It gets ugly really quickly.
but it's not the fault of xmlrpc, it's the implementation fault. there's other language out there that got a perfectly fine xmlrpc library, that doesn't assume anything about the transport.
worst come to worst, reimplementing XMLRPC is *easy* as long as the marshaling protocol library (XML) is available in a good quality, which is certainly the case in python.
The same hold true of JSonRPC, absolutely trivial to implement as long as you got the underlying JSon library.
Quoted 6 lines
Also, typing really hurts with dynamic languages because the type semantics never match up well which causes unexpected failures. A classic example with XML-RPC is None. There is no standard way to transport None in XML-RPC although there are common extensions to support it. Of course, not all clients support it and there isn't really a standard way to negotiate it. It's really a mess.
how does QMP handle None then ?
Quoted 3 lines
At this point, we have a close-to-mergable set of patches that provide real value. I don't see a reason not to merge them once they've been resubmitted.
it provides no real value at this point. it start providing real value when everything out (libvirt, xend, ..) there is using it.
script people won't be able to use QMP anyway, and probably still going to use the monitor as it is (you don't expect them to parse a line protocol with quoted string in bash do you ?)
Quoted 1 line
I don't believe an RPC is going to be a dramatic simplification.
it's not really a simplification, but QMP while it look simple now (just looking at the C side, without factoring the cost of reimplementing this text protocol in language that want to use it), it's becoming not simple.
already the protocol is growing quoted string, utf8 support, ...
What's next ? multi line protocol to extends it ?
IMAP grew probably the same way, as a simple protocol in the beggining, but look at the very useful description that Felip made (Thanks Felip btw)
it's like printing XML yourself with printf instead of using the outputing code of the library because it's "harder". you end up with some not correctly escaped XML, etc
Quoted 6 lines
It
could be if there were a high quality RPC out there that had well maintained client/server libraries. I don't believe such a thing exists though. If you want to keep arguing this, let's focus on actual RPC implementations. I think we've agreed that JSON-RPC isn't going to work because of the lack of client/server library availability.
So at this point your dismissing JSON-RPC just on the fact that there's no server/client implementation available right away, but you don't dismiss it on any other ground related to the protocol itself ? is that right ?
Quoted 4 lines
XML-RPC
falls into the same boat for C as far as I'm concerned. If there are any other ones you want to consider, let's look at them so that we can make progress in this discussion.
Signature · 1 line
Vincent
Quoted 3 lines · Anthony Liguori
The odd thing about it is you still have to parse json in JavaScript to avoid security problems. Parsing json in js seems to eliminate 99% of the useness of it.
I think this is mistaken. Last time I looked, you would check the whole string with a simple regexp to look for badness, and that confirms it's safe to eval the json. Good json always passes the regexp. Not that I've ever done any json :-)
-- Jamie
Quoted 1 line · Anthony Liguori
In 5 years, it will be shocking that qemu uses jsonrpc instead of the next super-neat RPC protocol.
But, we'll be able to control qemu from our browsers!
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 7 lines · Anthony Liguori
Quoted 4 lines · Avi Kivity
Quoted 2 lines
I'd rather not do json.
Why not?
Because it's too much complexity at this level in the stack.
It's not.
instead of replying the OK, ERROR .. you reply a JsonRPC OK|ERROR structure with the same string (quoted and utf-8) as the data payload.
instead of parsing each command line with the text protocol thing, you'll get every parameter from a json array of string (can contain any characters compared to QMP so far).
if you want you can make things generic at the marshalling level, by using string everywhere .. which is what QMP is doing.
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
As they stand, I think the current patch set is very close to being mergable. I'd rather not go off on a new tangent.
It fulfils all are requirements and more. There's a C implementation available. We can concentrate on the commands and ignore the plumbing.
Quoted 8 lines
The same is true for XML-RPC, CIM, dbus, etc. RPC grow like weeds. In 5 years, it will be shocking that qemu uses jsonrpc instead of the next super-neat RPC protocol.
If we did a real RPC mechanism, I'd want to stick with something tried-and-true like XDR. But an RPC mechanism is not the monitor. If we were going to do that, there's no point in morphing the monitor into that.
CIM, dbus are binary protocol. same apparently for XDR (which i've never used). only XMLRPC is fairly similar (but much more verbose and xml is ugly)
Jsonrpc is just one "standard" marshalling protocol, that I think would fit quite well, but the point is using something already out there not something that looks like the evil cousin of the IRC protocol.
Usually too, if the rpc has been correctly bolted on something, changing one to the other is usually trivial (for example xmlrpc to jsonrpc or the other way). So maybe in 5 years, people will push for a new protocol XYZ .. but that doesn't mean it's not going to happen to QMP too anyway...
Quoted 3 lines
What we're doing here is adding a computer-mode to the existing monitor. This will be easy for current apps to consume and for us to transition the code base to. It's a safe and incremental improvement.
i don't see my "proposal", as going against this statement. you could bolt jsonrpc or xmlrpc or <insert your known protocol> in an incremental fashion too.
Quoted 1 line
I don't want to invent Xen-API here and go through that pain in QEMU.
This is really *not* comparable to Xen-API. QEMU manage VM and only VM. Xen-API goes much further.
Signature · 1 line
Vincent
Quoted 7 lines · Luiz Capitulino
This file contains detailed QMP description and definitions.
Signed-off-by Luiz Capitulino <[email protected]>---
monitor-protocol-spec.txt | 180 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 180 insertions(+), 0 deletions(-)create mode 100644 monitor-protocol-spec.txt
I'd rather have the description in texi format and include it as an appendix from qemu-tech.texi.
PATCH 2/11 QMP: Introduce MONITOR_USE_CONTROL flag
In reply to a message not in the archiveIt will be used to tell Monitor we are in control mode, also introduce a function to do the check.
--- monitor.c | 6 ++++++ monitor.h | 1 + 2 files changed, 7 insertions(+), 0 deletions(-)
monitor.c +6 -0
@@ -93,6 +93,12 @@ Monitor *cur_mon = NULL;static void monitor_command_cb (Monitor *mon,const char *cmdline,void *opaque); +/* Return true if in control mode, false otherwise */ +static inline int monitor_ctrl_mode (Monitor *mon) +{ +return (mon->flags &MONITOR_USE_CONTROL ); +} +static void monitor_read_command (Monitor *mon,int show_prompt) {readline_start (mon->rs,"(qemu) " ,0 , monitor_command_cb,NULL );
monitor.h +1 -0
@@ -10,6 +10,7 @@ extern Monitor *cur_mon;/* flags for monitor_init */ #define MONITOR_IS_DEFAULT 0x01#define MONITOR_USE_READLINE 0x02 +#define MONITOR_USE_CONTROL 0x04void monitor_init (CharDriverState *chr,int flags);
Signature · 1 line
1.6.3.GIT
PATCH 3/11 QMP: Introduce protocol print functions
In reply to a message not in the archiveThis change introduces the functions that will be used by the Monitor to output data in the format defined by the protocol specification.
There are four functions:
o monitor_printf_bad() signals a protocol error o monitor_print_ok() signals successful execution o monitor_printf_err() used by commands, to signal execution errors
o monitor_printf_data() used by commands, to output data
For now monitor_print_ok() compilation is disabled to avoid breaking the build as the function is not currently used. It will be enabled by a later commit.
--- monitor.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ monitor.h | 3 ++ qemu-tool.c | 12 +++++++++++ 3 files changed, 78 insertions(+), 0 deletions(-)
monitor.c +63 -0
@@ -199,6 +199,69 @@ static int monitor_fprintf(FILE *stream, const char *fmt, ...)return 0 ; } +/* +* QEMU Monitor Control print functions +*/ + +/* Protocol errors */ +void monitor_printf_bad (Monitor *mon,const char *fmt, ...) +{ +if (monitor_ctrl_mode (mon)) { +va_list ap; + +monitor_puts (mon,"- BAD " ); + +va_start (ap, fmt); +monitor_vprintf (mon, fmt, ap); +va_end (ap); + } +} + +#if 0 +/* OK command completion, 'info' commands are special */ +static void monitor_print_ok (Monitor *mon,const char *cmd,const char *arg) +{ +if (!monitor_ctrl_mode (mon)) +return ; + +monitor_printf (mon,"+ OK %s" , cmd); +if (!strcmp (cmd,"info" )) +monitor_printf (mon," %s" , arg); +monitor_puts (mon," completed\n" ); +} +#endif + +static void monitor_ctrl_pprintf (Monitor *mon,const char *prefix, +const char *fmt,va_list ap) +{ +if (monitor_ctrl_mode (mon)) +monitor_puts (mon, prefix); + +monitor_vprintf (mon, fmt, ap); +} + +/* Should be used by commands to signal errors, will add the prefix +* '- ERR ' if in control mode */ +void monitor_printf_err (Monitor *mon,const char *fmt, ...) +{ +va_list ap; + +va_start (ap, fmt); +monitor_ctrl_pprintf (mon,"- ERR " , fmt, ap); +va_end (ap); +} + +/* Should be used by commands to print any output which is not +* an error, will add the prefix '= ' if in control mode */ +void monitor_printf_data (Monitor *mon,const char *fmt, ...) +{ +va_list ap; + +va_start (ap, fmt); +monitor_ctrl_pprintf (mon,"= " , fmt, ap); +va_end (ap); +} +static int compare_cmd (const char *name,const char *list) {const char *p, *pstart;
monitor.h +3 -0
@@ -24,6 +24,9 @@ void monitor_read_bdrv_key_start(Monitor *mon, BlockDriverState *bs,void monitor_vprintf (Monitor *mon,const char *fmt,va_list ap);void monitor_printf (Monitor *mon,const char *fmt, ...) __attribute__ ((__format__ (__printf__,2 ,3 ))); +void monitor_printf_bad (Monitor *mon,const char *fmt, ...); +void monitor_printf_err (Monitor *mon,const char *fmt, ...); +void monitor_printf_data (Monitor *mon,const char *fmt, ...);void monitor_print_filename (Monitor *mon,const char *filename);void monitor_flush (Monitor *mon);
qemu-tool.c +12 -0
@@ -40,6 +40,18 @@ void monitor_print_filename(Monitor *mon, const char *filename) { } +void monitor_printf_bad (Monitor *mon,const char *fmt, ...) +{ +} + +void monitor_printf_err (Monitor *mon,const char *fmt, ...) +{ +} + +void monitor_printf_data (Monitor *mon,const char *fmt, ...) +{ +} +QEMUBH *qemu_bh_new (QEMUBHFunc *cb,void *opaque) {QEMUBH *bh;
Signature · 1 line
1.6.3.GIT
Re: [Qemu-devel] [PATCH 03/11] QMP: Introduce protocol print functions
Quoted 100 lines · Luiz Capitulino
This change introduces the functions that will be used by the Monitor to output data in the format defined by the protocol specification.
There are four functions:
o monitor_printf_bad() signals a protocol error o monitor_print_ok() signals successful execution o monitor_printf_err() used by commands, to signal execution errorso
monitor_printf_data()used by commands, to output dataFor now
monitor_print_ok()compilation is disabled to avoid breaking the build as the function is not currently used. It will be enabled by a later commit.Signed-off-by Luiz Capitulino <[email protected]>--- monitor.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ monitor.h | 3 ++ qemu-tool.c | 12 +++++++++++ 3 files changed, 78 insertions(+), 0 deletions(-)monitor.c +63
-0@@ -199,6 +199,69 @@ static int monitor_fprintf(FILE *stream, const char *fmt, ...)return 0 ; } +/* +* QEMU Monitor Control print functions +*/ + +/* Protocol errors */ +void monitor_printf_bad (Monitor *mon,const char *fmt, ...) +{ +if (monitor_ctrl_mode (mon)) { +va_list ap; + +monitor_puts (mon,"- BAD " ); + +va_start (ap, fmt); +monitor_vprintf (mon, fmt, ap); +va_end (ap); + } +} + +#if 0 +/* OK command completion, 'info' commands are special */ +static void monitor_print_ok (Monitor *mon,const char *cmd,const char *arg) +{ +if (!monitor_ctrl_mode (mon)) +return ; + +monitor_printf (mon,"+ OK %s" , cmd); +if (!strcmp (cmd,"info" )) +monitor_printf (mon," %s" , arg); +monitor_puts (mon," completed\n" ); +} +#endif + +static void monitor_ctrl_pprintf (Monitor *mon,const char *prefix, +const char *fmt,va_list ap) +{ +if (monitor_ctrl_mode (mon)) +monitor_puts (mon, prefix); + +monitor_vprintf (mon, fmt, ap); +} + +/* Should be used by commands to signal errors, will add the prefix +* '- ERR ' if in control mode */ +void monitor_printf_err (Monitor *mon,const char *fmt, ...) +{ +va_list ap; + +va_start (ap, fmt); +monitor_ctrl_pprintf (mon,"- ERR " , fmt, ap); +va_end (ap); +} + +/* Should be used by commands to print any output which is not +* an error, will add the prefix '= ' if in control mode */ +void monitor_printf_data (Monitor *mon,const char *fmt, ...) +{ +va_list ap; + +va_start (ap, fmt); +monitor_ctrl_pprintf (mon,"= " , fmt, ap); +va_end (ap); +} +static int compare_cmd (const char *name,const char *list) {const char *p, *pstart;monitor.h +0
-0
monitor.h void monitor_read_bdrv_key_start(Monitor *mon, BlockDriverState *bs, · line 30
Quoted 7 lines
monitor.h +3
-0@@ -24,6 +24,9 @@ void monitor_read_bdrv_key_start(Monitor *mon, BlockDriverState *bs,void monitor_vprintf (Monitor *mon,const char *fmt,va_list ap);void monitor_printf (Monitor *mon,const char *fmt, ...) __attribute__ ((__format__ (__printf__,2 ,3 ))); +void monitor_printf_bad (Monitor *mon,const char *fmt, ...); +void monitor_printf_err (Monitor *mon,const char *fmt, ...); +void monitor_printf_data (Monitor *mon,const char *fmt, ...);
Probably could just introduce a flag to monitor_printf(). In fact, you could go kernel style and have:
monitor_printf(mon, MON_BAD "bad monitor command\n"); monitor_printf(mon, MON_DATA "some monitor data\n");
With MON_DATA being explicit.
If not, make sure to add the format attribute to the printfs.
Regards,
Anthony Liguori
Re: [Qemu-devel] [PATCH 03/11] QMP: Introduce protocol print functions
Quoted 111 lines · Anthony Liguori
Quoted 99 lines · Luiz Capitulino
This change introduces the functions that will be used by the Monitor to output data in the format defined by the protocol specification.
There are four functions:
o monitor_printf_bad() signals a protocol error o monitor_print_ok() signals successful execution o monitor_printf_err() used by commands, to signal execution errorso
monitor_printf_data()used by commands, to output dataFor now
monitor_print_ok()compilation is disabled to avoid breaking the build as the function is not currently used. It will be enabled by a later commit.Signed-off-by Luiz Capitulino <[email protected]>--- monitor.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ monitor.h | 3 ++ qemu-tool.c | 12 +++++++++++ 3 files changed, 78 insertions(+), 0 deletions(-)monitor.c +62
-0@@ -199,6 +199,69 @@ static int monitor_fprintf(FILE *stream, const char *fmt, ...)return 0 ; } +/* +* QEMU Monitor Control print functions +*/ + +/* Protocol errors */ +void monitor_printf_bad(Monitor *mon, const char *fmt, ...) +{ +if (monitor_ctrl_mode (mon)) { +va_list ap; + +monitor_puts (mon,"- BAD " ); + +va_start (ap, fmt); +monitor_vprintf (mon, fmt, ap); +va_end (ap); + } +} + +#if 0 +/* OK command completion, 'info' commands are special */ +static void monitor_print_ok(Monitor *mon, const char *cmd, const char *arg) +{ +if (!monitor_ctrl_mode (mon)) +return ; + +monitor_printf (mon,"+ OK %s" , cmd); +if (!strcmp (cmd,"info" )) +monitor_printf (mon," %s" , arg); +monitor_puts (mon," completed\n" ); +} +#endif + +static void monitor_ctrl_pprintf (Monitor *mon,const char *prefix, +const char *fmt,va_list ap) +{ +if (monitor_ctrl_mode (mon)) +monitor_puts (mon, prefix); + +monitor_vprintf (mon, fmt, ap); +} + +/* Should be used by commands to signal errors, will add the prefix +* '- ERR ' if in control mode */ +void monitor_printf_err (Monitor *mon,const char *fmt, ...) +{ +va_list ap; + +va_start (ap, fmt); +monitor_ctrl_pprintf (mon,"- ERR " , fmt, ap); +va_end (ap); +} + +/* Should be used by commands to print any output which is not +* an error, will add the prefix '= ' if in control mode */ +void monitor_printf_data (Monitor *mon,const char *fmt, ...) +{ +va_list ap; + +va_start (ap, fmt); +monitor_ctrl_pprintf (mon,"= " , fmt, ap); +va_end (ap); +} +static int compare_cmd (const char *name,const char *list) {const char *p, *pstart;monitor.h +0
-0monitor.h void monitor_read_bdrv_key_start(Monitor *mon, BlockDriverState *bs, · line 30
Quoted 7 lines
monitor.h +3
-0@@ -24,6 +24,9 @@ void monitor_read_bdrv_key_start(Monitor *mon, BlockDriverState *bs,void monitor_vprintf (Monitor *mon,const char *fmt,va_list ap);void monitor_printf (Monitor *mon,const char *fmt, ...) __attribute__ ((__format__ (__printf__,2 ,3 ))); +void monitor_printf_bad (Monitor *mon,const char *fmt, ...); +void monitor_printf_err (Monitor *mon,const char *fmt, ...); +void monitor_printf_data (Monitor *mon,const char *fmt, ...);Probably could just introduce a flag to
monitor_printf(). In fact, you could go kernel style and have:
Quoted 6 lines
monitor_printf(mon, MON_BAD "bad monitor command\n");monitor_printf(mon, MON_DATA "some monitor data\n");With
MON_DATAbeing explicit.If not, make sure to add the format attribute to the printfs.
I think different functions make sense because some messages may have additional attributes (not only a text message), that may be arguments to the function. e.g.:
void monitor_printf_event(Monitor *mon, qemu_event_t event, const char *fmt, ...);
void monitor_printf_err(Monitor *mon, int status_code, const char *fmt, ...);
(or maybe we can just call it monitor_printf_result(), being applicable to both success and failure, depending on the status code).
Signature · 1 line
Eduardo
PATCH 4/11 QMP: Make monitor_handle_command() QMP aware
In reply to a message not in the archiveTwo changes are needed in monitor_handle_command() so that it knows how to work in control mode.
First, it has to know whether a given command is part of the protocol or not. Second, it has to print the correct server response when a command execution finishes.
A good approach to do this would be:
1. Add a new member to struct mon_cmd_t, which will tell if the command is already part of the protocol.
2. Change handler_* functions to return exit status, so that monitor_handle_command() can print "+ OK" or "- ERR" accordling.
However, to make the prototype simpler I have chosen a quick & dirty approach, which is:
1. Introduce valid_control_cmd(). This function has a listing of commands that have already been ported and thus are part of the protocol.
2. Always print "+ OK", as current commands will probably not print "- ERR" :). This requires enabling monitor_print_ok() compilation.
--- monitor.c | 35 +++++++++++++++++++++++++++++++++-- 1 files changed, 33 insertions(+), 2 deletions(-)
monitor.c +33 -2
@@ -217,7 +217,6 @@ void monitor_printf_bad(Monitor *mon, const char *fmt, ...) } } -#if 0 /* OK command completion, 'info' commands are special */ static void monitor_print_ok (Monitor *mon,const char *cmd,const char *arg) {
@@ -229,7 +228,6 @@ static void monitor_print_ok(Monitor *mon, const char *cmd, const char *arg)monitor_printf (mon," %s" , arg);monitor_puts (mon," completed\n" ); } -#endif static void monitor_ctrl_pprintf (Monitor *mon,const char *prefix,const char *fmt,va_list ap)
@@ -2476,6 +2474,35 @@ static const char *get_command_name(const char *cmdline,return p; } +/* When in control mode, return true if 'cmd' is part of the protocol, +* return false otherwise */ +static int valid_control_cmd (Monitor *mon,const char *cmd, +const char *cmdline) +{ +int i; +const char *valid_cmds[] = {NULL }; +const char *valid_infos[] = {NULL }; + +if (!monitor_ctrl_mode (mon)) { +/* all commands are valid in user-mode mode */ +return 1 ; + } + +for (i =0 ; valid_cmds[i]; i++) +if (compare_cmd (valid_cmds[i], cmd)) +return 1 ; + +/* info is special */ +if (compare_cmd (cmd,"info" )) { +for (i =0 ; valid_infos[i]; i++) +if (strstr (cmdline, valid_infos[i])) +return 1 ; + } + +monitor_printf_bad (mon,"unknown command\n" ); +return 0 ; +} +static int default_fmt_format ='x' ;static int default_fmt_size =4 ;
@@ -2512,6 +2539,9 @@ static void monitor_handle_command(Monitor *mon, const char *cmdline)if (!p)return ; +if (valid_control_cmd (mon, cmdname, cmdline) ==0 ) +return ; +/* find the command */ for (cmd = mon_cmds; cmd->name !=NULL ; cmd++) {if (compare_cmd (cmdname, cmd->name))
@@ -2794,6 +2824,7 @@ static void monitor_handle_command(Monitor *mon, const char *cmdline)monitor_printf (mon,"unsupported number of arguments: %d\n" , nb_args); goto fail; } +monitor_print_ok (mon, cmdname, args[0 ]); fail:for (i =0 ; i <MAX_ARGS ; i++)qemu_free (str_allocated[i]);
Signature · 1 line
1.6.3.GIT
Re: [Qemu-devel] [PATCH 04/11] QMP: Make monitor_handle_command() QMP aware
Quoted 7 lines · Luiz Capitulino
Two changes are needed in
monitor_handle_command()so that it knows how to work in control mode.First, it has to know whether a given command is part of the protocol or not. Second, it has to print the correct server response when a command execution finishes.
Quoted 9 lines
A good approach to do this would be:
1. Add a new member to struct
mon_cmd_t, which will tell if the command is already part of the protocol.2. Change handler_* functions to return exit status, so that
monitor_handle_command()can print "+ OK" or "- ERR" accordling.
These changes are a requirement for merging.
Regards,
Anthony Liguori
Re: [Qemu-devel] [PATCH 04/11] QMP: Make monitor_handle_command() QMP aware
Quoted 10 lines · On Tue, 23 Jun 2009 08:51:16 -0500 Anthony Liguori <[email protected]>…
Quoted 9 lines · Luiz Capitulino
Two changes are needed in
monitor_handle_command()so that it knows how to work in control mode.First, it has to know whether a given command is part of the protocol or not. Second, it has to print the correct server response when a command execution finishes.
A good approach to do this would be:
Quoted 8 lines
Quoted 7 lines
1. Add a new member to struct
mon_cmd_t, which will tell if the command is already part of the protocol.2. Change handler_* functions to return exit status, so that
monitor_handle_command()can print "+ OK" or "- ERR" accordling.These changes are a requirement for merging.
Yes, sure. Would never ask you to merge such a hack.
PATCH 5/11 QMP: Introduce control mode chardev handling
In reply to a message not in the archiveControl mode disables readline support and handles Monitor's chardev by itself.
In order to this this change adds:
o MonitorControl type: used to store input from the client
o monitor_control_read(): read input from the chardev
o monitor_control_event(): prints our greeting message
Support to control mode is also added to monitor_init().
Also note that monitor_control_read() is a bit simple right now, will be improved in future versions.
--- monitor.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 files changed, 49 insertions(+), 2 deletions(-)
monitor.c +49 -2
@@ -70,6 +70,11 @@ typedef struct mon_cmd_t {const char *help; } mon_cmd_t; +typedef struct MonitorControl { +uint8_t buf[128 ]; +int size; +}MonitorControl ; +struct Monitor {CharDriverState *chr;int flags;
@@ -77,6 +82,7 @@ struct Monitor {uint8_t outbuf[1024 ];int outbuf_index;ReadLineState *rs; +MonitorControl *mc;CPUState *mon_cpu;BlockDriverCompletionFunc *password_completion_cb;void *password_opaque;
@@ -3038,6 +3044,31 @@ static int monitor_can_read(void *opaque)return (mon->suspend_cnt ==0 ) ?128 :0 ; } +static void monitor_control_read (void *opaque,const uint8_t *buf,int size) +{ +Monitor *old_mon = cur_mon; +int i; + + cur_mon = opaque; + +for (i =0 ; i < size; i++) { +if (buf[i] =='\r' || buf[i] =='\n' ) { + cur_mon->mc->buf[cur_mon->mc->size] ='\0' ; + cur_mon->mc->size =0 ; +monitor_handle_command (cur_mon, (char *)cur_mon->mc->buf); + }else { + cur_mon->mc->buf[cur_mon->mc->size++] = buf[i]; +if (cur_mon->mc->size ==sizeof (cur_mon->mc->buf)) { +monitor_printf_bad (cur_mon,"command too long\n" ); + cur_mon->mc->size =0 ; +return ; + } + } + } + + cur_mon = old_mon; +} +static void monitor_read (void *opaque,const uint8_t *buf,int size) {Monitor *old_mon = cur_mon;
@@ -3081,6 +3112,15 @@ void monitor_resume(Monitor *mon)readline_show_prompt (mon->rs); } +static void monitor_control_event (void *opaque,int event) +{ +if (event ==CHR_EVENT_RESET ) { +Monitor *mon = opaque; + mon->mc->size =0 ; +monitor_printf (mon,"+ QEMU %s QMP 0.1\n" ,QEMU_VERSION ); + } +} +static void monitor_event (void *opaque,int event) {Monitor *mon = opaque;
@@ -3138,8 +3178,15 @@ void monitor_init(CharDriverState *chr, int flags)monitor_read_command (mon,0 ); } -qemu_chr_add_handlers (chr, monitor_can_read, monitor_read, monitor_event, - mon); +if (monitor_ctrl_mode (mon)) { + mon->mc =qemu_mallocz (sizeof (MonitorControl)); +/* Control mode requires special handlers */ +qemu_chr_add_handlers (chr, monitor_can_read, monitor_control_read, + monitor_control_event, mon); + }else { +qemu_chr_add_handlers (chr, monitor_can_read, monitor_read, + monitor_event, mon); + }LIST_INSERT_HEAD (&mon_list, mon, entry);if (!cur_mon || (flags &MONITOR_IS_DEFAULT ))
Signature · 1 line
1.6.3.GIT
PATCH 6/11 QMP: Introduce asynchronous events infrastructure
In reply to a message not in the archiveIt is just an exported function that will be used by other subsystems to print specific events to the output buffer.
This is based in ideas by Amit Shah <[email protected]>.
--- monitor.c | 18 ++++++++++++++++++ monitor.h | 6 ++++++ qemu-tool.c | 4 ++++ 3 files changed, 28 insertions(+), 0 deletions(-)
monitor.c +18 -0
@@ -266,6 +266,24 @@ void monitor_printf_data(Monitor *mon, const char *fmt, ...)va_end (ap); } +/* Asynchronous events main function */ +void monitor_notify_event (MonitorEvent event) +{ +if (!monitor_ctrl_mode (cur_mon)) +return ; + +assert (event <EVENT_MAX ); +monitor_puts (cur_mon,"* EVENT " ); + +switch (event) { +case EVENT_MAX : +// Avoid gcc warning, will never get here +break ; + } + +monitor_puts (cur_mon,"\n" ); +} +static int compare_cmd (const char *name,const char *list) {const char *p, *pstart;
monitor.h +6 -0
@@ -12,6 +12,11 @@ extern Monitor *cur_mon;#define MONITOR_USE_READLINE 0x02#define MONITOR_USE_CONTROL 0x04 +/* QMP events */ +typedef enum MonitorEvent { +EVENT_MAX , +}MonitorEvent ; +void monitor_init (CharDriverState *chr,int flags);int monitor_suspend (Monitor *mon);
@@ -28,6 +33,7 @@ void monitor_printf_bad(Monitor *mon, const char *fmt, ...);void monitor_printf_err (Monitor *mon,const char *fmt, ...);void monitor_printf_data (Monitor *mon,const char *fmt, ...);void monitor_print_filename (Monitor *mon,const char *filename); +void monitor_notify_event (MonitorEvent event);void monitor_flush (Monitor *mon);#endif /* !MONITOR_H */
qemu-tool.c +4 -0
@@ -40,6 +40,10 @@ void monitor_print_filename(Monitor *mon, const char *filename) { } +void monitor_notify_event (MonitorEvent event) +{ +} +void monitor_printf_bad (Monitor *mon,const char *fmt, ...) { }
Signature · 1 line
1.6.3.GIT
[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure
Quoted 28 lines · Luiz Capitulino
It is just an exported function that will be used by other subsystems to print specific events to the output buffer.
This is based in ideas by Amit Shah <[email protected]>.
Signed-off-by Luiz Capitulino <[email protected]>--- monitor.c | 18 ++++++++++++++++++ monitor.h | 6 ++++++ qemu-tool.c | 4 ++++ 3 files changed, 28 insertions(+), 0 deletions(-)monitor.c +8
-0@@ -266,6 +266,24 @@ void monitor_printf_data(Monitor *mon, const char *fmt, ...)va_end (ap); } +/* Asynchronous events main function */ +void monitor_notify_event (MonitorEvent event) +{ +if (!monitor_ctrl_mode (cur_mon)) +return ; + +assert (event <EVENT_MAX ); +monitor_puts (cur_mon,"* EVENT " );
monitor.c void monitor_printf_data(Monitor *mon, const char *fmt, ...) · line 286
Quoted 10 lines
monitor.c +10
-0+ +switch (event) { +case EVENT_MAX : +// Avoid gcc warning, will never get here +break ; + } + +monitor_puts (cur_mon,"\n" ); +} +
You shouldn't use cur_mon here. It's for legacy code that still assumes there can be only one monitor terminal. Instead, iterate over all monitors and distribute the event to those that are in control mode.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure
Quoted 35 lines · On Tue, 23 Jun 2009 10:59:27 +0200 Jan Kiszka <[email protected]> wrote:
Quoted 34 lines · Luiz Capitulino
It is just an exported function that will be used by other subsystems to print specific events to the output buffer.
This is based in ideas by Amit Shah <[email protected]>.
Signed-off-by Luiz Capitulino <[email protected]>--- monitor.c | 18 ++++++++++++++++++ monitor.h | 6 ++++++ qemu-tool.c | 4 ++++ 3 files changed, 28 insertions(+), 0 deletions(-)monitor.c +14
-0@@ -266,6 +266,24 @@ void monitor_printf_data(Monitor *mon, const char *fmt, ...)va_end (ap); } +/* Asynchronous events main function */ +void monitor_notify_event (MonitorEvent event) +{ +if (!monitor_ctrl_mode (cur_mon)) +return ; + +assert (event <EVENT_MAX ); +monitor_puts (cur_mon,"* EVENT " ); + +switch (event) { +case EVENT_MAX : +// Avoid gcc warning, will never get here +break ; + }
Quoted 8 lines
monitor.c void monitor_printf_data(Monitor *mon, const char *fmt, ...) · line 286
Quoted 4 lines
monitor.c +4
-0+ +monitor_puts (cur_mon,"\n" ); +} +You shouldn't use
cur_monhere. It's for legacy code that still assumes there can be only one monitor terminal. Instead, iterate over all monitors and distribute the event to those that are in control mode.
Thanks. I was really in doubt on what to do here. :)
[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure
In reply to Luiz CapitulinoQuoted 6 lines · Luiz Capitulino
It is just an exported function that will be used by other subsystems to print specific events to the output buffer.
This is based in ideas by Amit Shah <[email protected]>.
Signed-off-by Luiz Capitulino <[email protected]>
As per my mail to the lead message in this thread, I'm thinking it might be valuable to include a timestamp with all events, and command replies. A 64-bit seconds-since-epoch prefix on all replies, no need for a human-format timestamp
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure
Quoted 7 lines · Daniel P. Berrange
Quoted 5 lines · Luiz Capitulino
It is just an exported function that will be used by other subsystems to print specific events to the output buffer.
This is based in ideas by Amit Shah<[email protected]>.
Quoted 8 lines
Quoted 2 lines
Signed-off-by Luiz Capitulino<[email protected]>As per my mail to the lead message in this thread, I'm thinking it might be valuable to include a timestamp with all events, and command replies. A 64-bit seconds-since-epoch prefix on all replies, no need for a human-format timestamp
A bit of an overkill... seconds.nanoseconds.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure
In reply to Daniel P. BerrangeQuoted 8 lines · Daniel P. Berrange
Quoted 2 lines · Luiz Capitulino
It is just an exported function that will be used by other subsystems to print specific events to the output buffer.
As per my mail to the lead message in this thread, I'm thinking it might be valuable to include a timestamp with all events, and command replies. A 64-bit seconds-since-epoch prefix on all replies, no need for a human-format timestamp
I'm reluctant to add timestamps just because they seem like a cool feature. I think it's important to first understand why you want timestamps, and exactly where the timestamp comes from. What are you expecting to gain from recording the timestamp in qemu rather than the management application?
I'd expect nearest-second timestamps to be pretty much useless for any practical purpose.
Also, should the timestamps be real time or virtual time?
Paul
[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure
In reply to Luiz CapitulinoQuoted 28 lines · Luiz Capitulino
It is just an exported function that will be used by other subsystems to print specific events to the output buffer.
This is based in ideas by Amit Shah <[email protected]>.
Signed-off-by Luiz Capitulino <[email protected]>--- monitor.c | 18 ++++++++++++++++++ monitor.h | 6 ++++++ qemu-tool.c | 4 ++++ 3 files changed, 28 insertions(+), 0 deletions(-)monitor.c +8
-0@@ -266,6 +266,24 @@ void monitor_printf_data(Monitor *mon, const char *fmt, ...)va_end (ap); } +/* Asynchronous events main function */ +void monitor_notify_event (MonitorEvent event) +{ +if (!monitor_ctrl_mode (cur_mon)) +return ; + +assert (event <EVENT_MAX ); +monitor_puts (cur_mon,"* EVENT " );
monitor.c void monitor_printf_data(Monitor *mon, const char *fmt, ...) · line 286
Quoted 10 lines
monitor.c +10
-0+ +switch (event) { +case EVENT_MAX : +// Avoid gcc warning, will never get here +break ; + } + +monitor_puts (cur_mon,"\n" ); +} +
If a client is not reading from the monitor channel quickly enough, then would this cause QEMU to block once the FD buffer is full ? A QEMU level buffer might give more leyway, but we don't want to grow unbounded, so ultimately we'll end up having to drop events. At which point you'd want to send an event to the client indicating that the event queue overflowed, so it can take remedial action to re-sync its state.
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure
Quoted 18 lines · Daniel P. Berrange
Quoted 18 lines
monitor.c +18
-0+/* Asynchronous events main function */ +void monitor_notify_event (MonitorEvent event) +{ +if (!monitor_ctrl_mode (cur_mon)) +return ; + +assert (event<EVENT_MAX ); +monitor_puts (cur_mon,"* EVENT " ); + +switch (event) { +case EVENT_MAX : +// Avoid gcc warning, will never get here +break ; + } + +monitor_puts (cur_mon,"\n" ); +} +
Quoted 9 lines
Quoted 1 line
If a client is not reading from the monitor channel quickly enough, then would this cause QEMU to block once the FD buffer is full ? A QEMU level buffer might give more leyway, but we don't want to grow unbounded, so ultimately we'll end up having to drop events. At which point you'd want to send an event to the client indicating that the event queue overflowed, so it can take remedial action to re-sync its state.
IMO relying on kernel buffers is sufficient here. An 8k buffer will hold hundreds of events. If the system is so congested that management cannot consume events rapidly enough, then it will also be so congested that guest vcpus will be starved.
Blocking on buffers full is acceptable IMO.
Signature · 1 line
error compiling committee.c: too many arguments to function
[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure
In reply to Daniel P. BerrangeQuoted 38 lines · On Tue, 23 Jun 2009 11:32:51 +0100 "Daniel P. Berrange" <[email protected]>…
Quoted 37 lines · Luiz Capitulino
It is just an exported function that will be used by other subsystems to print specific events to the output buffer.
This is based in ideas by Amit Shah <[email protected]>.
Signed-off-by Luiz Capitulino <[email protected]>--- monitor.c | 18 ++++++++++++++++++ monitor.h | 6 ++++++ qemu-tool.c | 4 ++++ 3 files changed, 28 insertions(+), 0 deletions(-)monitor.c +17
-0@@ -266,6 +266,24 @@ void monitor_printf_data(Monitor *mon, const char *fmt, ...)va_end (ap); } +/* Asynchronous events main function */ +void monitor_notify_event (MonitorEvent event) +{ +if (!monitor_ctrl_mode (cur_mon)) +return ; + +assert (event <EVENT_MAX ); +monitor_puts (cur_mon,"* EVENT " ); + +switch (event) { +case EVENT_MAX : +// Avoid gcc warning, will never get here +break ; + } + +monitor_puts (cur_mon,"\n" ); +}
Quoted 8 lines
monitor.c void monitor_printf_data(Monitor *mon, const char *fmt, ...) · line 286
Quoted 1 line
monitor.c +1
-0+If a client is not reading from the monitor channel quickly enough, then would this cause QEMU to block once the FD buffer is full ? A QEMU level buffer might give more leyway, but we don't want to grow unbounded, so ultimately we'll end up having to drop events. At which point you'd want to send an event to the client indicating that the event queue overflowed, so it can take remedial action to re-sync its state.
Wouldn't a buffer flush right at the beginning of this function solve this?
At least for QEMU?
I'm not sure if it's possible to get QEMU blocked.
[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure
Quoted 50 lines · Luiz Capitulino
Quoted 38 lines · On Tue, 23 Jun 2009 11:32:51 +0100 "Daniel P. Berrange" <[email protected]>…
Quoted 37 lines · Luiz Capitulino
It is just an exported function that will be used by other subsystems to print specific events to the output buffer.
This is based in ideas by Amit Shah <[email protected]>.
Signed-off-by Luiz Capitulino <[email protected]>--- monitor.c | 18 ++++++++++++++++++ monitor.h | 6 ++++++ qemu-tool.c | 4 ++++ 3 files changed, 28 insertions(+), 0 deletions(-)monitor.c +17
-0@@ -266,6 +266,24 @@ void monitor_printf_data(Monitor *mon, const char *fmt, ...)va_end (ap); } +/* Asynchronous events main function */ +void monitor_notify_event (MonitorEvent event) +{ +if (!monitor_ctrl_mode (cur_mon)) +return ; + +assert (event <EVENT_MAX ); +monitor_puts (cur_mon,"* EVENT " ); + +switch (event) { +case EVENT_MAX : +// Avoid gcc warning, will never get here +break ; + } + +monitor_puts (cur_mon,"\n" ); +}Quoted 8 lines
monitor.c void monitor_printf_data(Monitor *mon, const char *fmt, ...) · line 286
Quoted 1 line
monitor.c +1
-0+If a client is not reading from the monitor channel quickly enough, then would this cause QEMU to block once the FD buffer is full ? A QEMU level buffer might give more leyway, but we don't want to grow unbounded, so ultimately we'll end up having to drop events. At which point you'd want to send an event to the client indicating that the event queue overflowed, so it can take remedial action to re-sync its state.
Quoted 2 lines
Wouldn't a buffer flush right at the beginning of this function solve this?
At least for QEMU?
No, if the messages aren't being consumed fast enough by the other side. If the consumer isn't consuming the messages fast enough, we would need to either drop messages or block on the channel (that's the code code above would do: block on write()).
I agree with Avi: if the system is so busy that the client doesn't consume the monitor messages fast enough, we are already screwed.
Signature · 1 line
Eduardo
[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure
Quoted 52 lines · Eduardo Habkost
Quoted 48 lines · Luiz Capitulino
Quoted 37 lines · On Tue, 23 Jun 2009 11:32:51 +0100 "Daniel P. Berrange" <[email protected]>…
Quoted 36 lines · Luiz Capitulino
It is just an exported function that will be used by other subsystems to print specific events to the output buffer.
This is based in ideas by Amit Shah <[email protected]>.
Signed-off-by Luiz Capitulino <[email protected]>--- monitor.c | 18 ++++++++++++++++++ monitor.h | 6 ++++++ qemu-tool.c | 4 ++++ 3 files changed, 28 insertions(+), 0 deletions(-)monitor.c +16
-0@@ -266,6 +266,24 @@ void monitor_printf_data(Monitor *mon, const char *fmt, ...)va_end (ap); } +/* Asynchronous events main function */ +void monitor_notify_event (MonitorEvent event) +{ +if (!monitor_ctrl_mode (cur_mon)) +return ; + +assert (event <EVENT_MAX ); +monitor_puts (cur_mon,"* EVENT " ); + +switch (event) { +case EVENT_MAX : +// Avoid gcc warning, will never get here +break ; + } + +monitor_puts (cur_mon,"\n" );Quoted 8 lines
monitor.c void monitor_printf_data(Monitor *mon, const char *fmt, ...) · line 286
Quoted 2 lines
monitor.c +2
-0+} +If a client is not reading from the monitor channel quickly enough, then would this cause QEMU to block once the FD buffer is full ? A QEMU level buffer might give more leyway, but we don't want to grow unbounded, so ultimately we'll end up having to drop events. At which point you'd want to send an event to the client indicating that the event queue overflowed, so it can take remedial action to re-sync its state.
Quoted 2 lines
Wouldn't a buffer flush right at the beginning of this function solve this?
At least for QEMU?
Quoted 7 lines
No, if the messages aren't being consumed fast enough by the other side. If the consumer isn't consuming the messages fast enough, we would need to either drop messages or block on the channel (that's the code code above would do: block on
write()).I agree with Avi: if the system is so busy that the client doesn't consume the monitor messages fast enough, we are already screwed.
Can't mgmt app and qemu also run on different hosts?
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure
Quoted 3 lines · Jan Kiszka
Can't mgmt app and qemu also run on different hosts?
In theory, yes. In practice you need someone to launch qemu and wait for it if it dies, so you'll have a daemon on the host. Typically it will be an intermediate daemon proxying all the guests for the real management server somewhere else.
Signature · 1 line
error compiling committee.c: too many arguments to function
[Qemu-devel] Re: [PATCH 06/11] QMP: Introduce asynchronous events infrastructure
In reply to Jan KiszkaQuoted 20 lines · Jan Kiszka
Quoted 19 lines · Eduardo Habkost
Quoted 4 lines · Luiz Capitulino
Quoted 1 line · On Tue, 23 Jun 2009 11:32:51 +0100 "Daniel P. Berrange" <[email protected]>…
Quoted 1 line
+
Quoted 8 lines
Quoted 6 lines
If a client is not reading from the monitor channel quickly enough, then would this cause QEMU to block once the FD buffer is full ? A QEMU level buffer might give more leyway, but we don't want to grow unbounded, so ultimately we'll end up having to drop events. At which point you'd want to send an event to the client indicating that the event queue overflowed, so it can take remedial action to re-sync its state.
Wouldn't a buffer flush right at the beginning of this function solve this?
At least for QEMU?No, if the messages aren't being consumed fast enough by the other side. If the consumer isn't consuming the messages fast enough, we would need to either drop messages or block on the channel (that's the code code above would do: block on
write()).
Quoted 4 lines
Quoted 2 lines
I agree with Avi: if the system is so busy that the client doesn't consume the monitor messages fast enough, we are already screwed.
Can't mgmt app and qemu also run on different hosts?
If you happened to run the monitor using a TCP backend, then yes the mgmt app could be remote. This would be just insane though, since there's no security there whatsoever. I don't think the QEMU monitor wants to be in the business of encrypting, authenticating & authorizing monitor clients. Rather it should just be local only, and any mgmt tool using QEMU can provide secure remote mgmt transports at a higher level.
Regards,
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
PATCH 7/11 QMP: Enable simple commands
In reply to a message not in the archive'cont', 'quit', 'stop', 'system_reset' and 'system_powerdown' are so simple that enabling them in valid_control_cmd() is the only step requered to port them.
--- monitor.c | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-)
monitor.c +6 -1
@@ -2504,7 +2504,12 @@ static int valid_control_cmd(Monitor *mon, const char *cmd, const char *cmdline) {int i; -const char *valid_cmds[] = {NULL }; +const char *valid_cmds[] = {"quit" , +"cont" , +"stop" , +"system_reset" , +"system_powerdown" , +NULL };const char *valid_infos[] = {NULL };if (!monitor_ctrl_mode (mon)) {
Signature · 1 line
1.6.3.GIT
PATCH 8/11 QMP: Port balloon command
In reply to a message not in the archive--- monitor.c | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-)
monitor.c +6 -4
@@ -1662,12 +1662,12 @@ static void do_info_balloon(Monitor *mon) actual =qemu_balloon_status ();if (kvm_enabled () && !kvm_has_sync_mmu ()) -monitor_printf (mon,"Using KVM without synchronous MMU, " +monitor_printf_err (mon,"Using KVM without synchronous MMU, " "ballooning disabled\n" );else if (actual ==0 ) -monitor_printf (mon,"Ballooning not activated in VM\n" ); +monitor_printf_err (mon,"Ballooning not activated in VM\n" );else -monitor_printf (mon,"balloon: actual=%d\n" , (int )(actual >>20 )); +monitor_printf_data (mon,"balloon: actual=%d\n" , (int )(actual >>20 )); } static void do_acl(Monitor *mon,
@@ -2509,8 +2509,10 @@ static int valid_control_cmd(Monitor *mon, const char *cmd,"stop" ,"system_reset" ,"system_powerdown" , +"balloon" , + NULL }; +const char *valid_infos[] = {"balloon" ,NULL }; -const char *valid_infos[] = {NULL };if (!monitor_ctrl_mode (mon)) {/* all commands are valid in user-mode mode */
Signature · 1 line
1.6.3.GIT
[Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
monitor.c static void do_info_balloon(Monitor *mon) · line 1671
Quoted 8 lines · Luiz Capitulino
else if (actual == 0)monitor.c +2
-2-monitor_printf (mon,"Ballooning not activated in VM\n" ); +monitor_printf_err (mon,"Ballooning not activated in VM\n" ); else - monitor_printf(mon, "balloon: actual=%d\n", (int)(actual>> 20)); + monitor_printf_data(mon, "balloon: actual=%d\n", (int)(actual>> 20)); }
Control mode should always use bytes and seconds (and this should be described in the spec). You avoid rounding, and more importantly, ambiguity and a source of unit conversion errors.
Patched that add a command to machine mode without updating the spec should be automatically NACKed.
We also need a way to discover that the command is available:
info commands
= info
= balloon
+ OK go for it
info command balloon
+ OK (no extensions yet)
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
Quoted 12 lines · Avi Kivity
Quoted 10 lines · Luiz Capitulino
else if (actual == 0)monitor.c +1
-2-monitor_printf (mon,"Ballooning not activated in VM\n" ); +monitor_printf_err (mon,"Ballooning not activated in VM\n" ); else - monitor_printf(mon, "balloon: actual=%d\n", (int)(actual>>20));
+ monitor_printf_data(mon, "balloon: actual=%d\n",
(int)(actual>> 20));
}
Quoted 1 line
Control mode should always use bytes and seconds (and this should be described in the spec). You avoid rounding, and more importantly, ambiguity and a source of unit conversion errors.
I'd actually like to see a lot more structure in this sort of output. For instance:
monitor_printf_list(mon, "balloon",
"actual", MON_TIME, actual, NULL);
How this gets output can then be conditional on control mode vs. human mode. In human mode, we can use human-friendly units like MBs. In control mode, we would always use bytes.
Quoted 3 lines
Patched that add a command to machine mode without updating the spec should be automatically NACKed.
We also need a way to discover that the command is available:
I think we want to version each command too.
Regards,
Anthony Liguori
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
Quoted 10 lines · Anthony Liguori
Quoted 1 line
Control mode should always use bytes and seconds (and this should be described in the spec). You avoid rounding, and more importantly, ambiguity and a source of unit conversion errors.
I'd actually like to see a lot more structure in this sort of output. For instance:
monitor_printf_list(mon, "balloon",
"actual",MON_TIME, actual, NULL);How this gets output can then be conditional on control mode vs. human mode. In human mode, we can use human-friendly units like MBs. In control mode, we would always use bytes.
That actually works well with rpc. You have a structure that contains the data. For machine mode, you send that away. For humans, you unit-convert, format and print.
Quoted 6 lines
Quoted 4 lines
Patched that add a command to machine mode without updating the spec should be automatically NACKed.
We also need a way to discover that the command is available:
I think we want to version each command too.
Not version, discover supported features.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
Quoted 8 lines · Avi Kivity
Quoted 6 lines
Quoted 4 lines
Patched that add a command to machine mode without updating the spec should be automatically NACKed.
We also need a way to discover that the command is available:
I think we want to version each command too.
Not version, discover supported features.
That's fine too. capabilities and versions are equivalent AFAICT so as long as one or the other is supported for each command, I'm happy.
Regards,
Anthony Liguori
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
In reply to Anthony LiguoriQuoted 20 lines · On Tue, 23 Jun 2009 08:59:42 -0500 Anthony Liguori <[email protected]>…
Quoted 12 lines · Avi Kivity
Quoted 10 lines · Luiz Capitulino
else if (actual == 0)monitor.c +1
-2-monitor_printf (mon,"Ballooning not activated in VM\n" ); +monitor_printf_err (mon,"Ballooning not activated in VM\n" ); else - monitor_printf(mon, "balloon: actual=%d\n", (int)(actual>>20));
+ monitor_printf_data(mon, "balloon: actual=%d\n",
(int)(actual>> 20));
}Quoted 3 lines
Control mode should always use bytes and seconds (and this should be described in the spec). You avoid rounding, and more importantly, ambiguity and a source of unit conversion errors.
I'd actually like to see a lot more structure in this sort of output. For instance:
Quoted 7 lines
monitor_printf_list(mon, "balloon",
"actual",MON_TIME, actual, NULL);How this gets output can then be conditional on control mode vs. human mode. In human mode, we can use human-friendly units like MBs. In control mode, we would always use bytes.
For me, this is one of the hardest aspects of the project.
It is more general than unit conversion, the problem is: how should
commands output (specially multline) look like?
I have talked with Eduardo about this, and I have chosen the simplest
approach, which is prefixing commands output by '='. That is, the protocol doesn't change commands output, just adds prefixes.
This approach has the advantage of making the protocol merge very
easy and I believe it will be less painful to maintain as well.
On the other hand, the format of most commands are not good/meant
to be parsed, this gives the possibility of:
Quoted 3 lines
Quoted 1 line
We also need a way to discover that the command is available:
I think we want to version each command too.
making it quite complicated. :)
If we go this way (of defining an output for each command or even
defining a version for each command) I fear we will take forever to get in agreement and merge the whole series.
Another related question is: should we port 100% of the command
set up front or can we merge small sets, defining a new version of the protocol each time?
What I would like to do for QMP V1 is:
1. Keep what this series is doing, that is, keep commands output unchanged and only add prefixes (unit conversion is ok though)
2. Define a set of commands to be part of V1, and port them
We can try to port the whole set as fast as we can (say, until
V3). Right after that, we could start improving commands output.
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
Quoted 7 lines · Luiz Capitulino
Quoted 2 lines · On Tue, 23 Jun 2009 08:59:42 -0500 Anthony Liguori <[email protected]>…
How this gets output can then be conditional on control mode vs. human mode. In human mode, we can use human-friendly units like MBs. In control mode, we would always use bytes.
Quoted 9 lines
For me, this is one of the hardest aspects of the project.
It is more general than unit conversion, the problem is: how should
commands output (specially multline) look like?I have talked with Eduardo about this, and I have chosen the simplest
approach, which is prefixing commands output by '='. That is, the protocol doesn't change commands output, just adds prefixes.
I'm not saying you should change command output. Most commands follow a similar format quite intentionally. That's:
<header>: <param>=<value>[,<param>=<value>[...]]
I'm just suggesting that you codify that instead of leaving it open coded everywhere. Nothing more than that.
Quoted 3 lines
This approach has the advantage of making the protocol merge very
easy and I believe it will be less painful to maintain as well.
FWIW, you could introduce monitor_printf_list() as your first patch.
Quoted 17 lines
Quoted 4 lines
Quoted 2 lines
We also need a way to discover that the command is available:
I think we want to version each command too.
making it quite complicated. :)
If we go this way (of defining an output for each command or even
defining a version for each command) I fear we will take forever to get in agreement and merge the whole series.Another related question is: should we port 100% of the command
set up front or can we merge small sets, defining a new version of the protocol each time?What I would like to do for QMP V1 is:
Quoted 8 lines
1. Keep what this series is doing, that is, keep commands output unchanged and only add prefixes (unit conversion is ok though)
2. Define a set of commands to be part of V1, and port them
We can try to port the whole set as fast as we can (say, until
V3). Right after that, we could start improving commands output.
I wanted to respond to the top level, but here's what I see as the potential merge plan:
1) Make all commands return a value
2) Add a mechanism to mark commands as being "structured"
3) Get the structured printfs in order.
4) Merge
I won't want QMP to be exposed as a usable interface until all commands are converted. This means holding off on the last patch I think. I don't think we'll get QMP for 0.11. I think it's likely going to be a 0.12 feature. However, I'd like to start merging this series as soon as humanly possible. I think it's already pretty close.
Signature · 3 lines
Regards, Anthony Liguori
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
Quoted 5 lines · Anthony Liguori
I wanted to respond to the top level, but here's what I see as the potential merge plan:
1) Make all commands return a value
2) Add a mechanism to mark commands as being "structured"
This seems like a good idea.
My original intent was to make the monitor interface a library. Various qemu users can link it with their favorite option: json/dbus/rpm/xml/soap ;)
So, we should first harden up the monitor commands like 1),2), and refactor the current monitor interface as a library above it.
A second or parallel phase is to implement a machine readable interface library.
Quoted 5 lines
3) Get the structured printfs in order.
4) MergeI won't want QMP to be exposed as a usable interface until all commands are converted. This means holding off on the last patch I think. I don't think we'll get QMP for 0.11. I think it's likely going to be a 0.12 feature. However, I'd like to start merging this series as soon as humanly possible. I think it's already pretty close.
If we push 'monitor standardization' into 0.11, the version will support future implementations of the protocol.
Regards,
Dor
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
Quoted 5 lines · On Thu, 25 Jun 2009 14:27:05 +0300 Dor Laor <[email protected]> wrote:
Quoted 4 lines · Anthony Liguori
I wanted to respond to the top level, but here's what I see as the potential merge plan:
Quoted 7 lines
Quoted 2 lines
1) Make all commands return a value
2) Add a mechanism to mark commands as being "structured"This seems like a good idea.
My original intent was to make the monitor interface a library. Various qemu users can link it with their favorite option:
json/dbus/rpm/xml/soap ;)
Yes, having a library was suggested by Amit some months ago. The
problem is that it has various issues wrt maintainability.
For example, libvirt is able to run two instances of different
versions of qemu at the same time. How to handle this if you update libmonitor.so?
I think people have agreed that having a protocol is good here, but
as Anthonhy puts it we need:
1. Choose between QMP and some kind of RPC
2. If RPC is choosen, what kind of RPC to use
Quoted 3 lines
So, we should first harden up the monitor commands like 1),2), and refactor the current
monitor interface as a library above it.
If I'm not mistaken, "structured" commands only makes sense with
QMP. The plan will be different if we choose RPC.
Quoted 15 lines
Quoted 9 lines
3) Get the structured printfs in order.
4) MergeI won't want QMP to be exposed as a usable interface until all commands are converted. This means holding off on the last patch I think. I don't think we'll get QMP for 0.11. I think it's likely going to be a 0.12 feature. However, I'd like to start merging this series as soon as humanly possible. I think it's already pretty close.
If we push 'monitor standardization' into 0.11, the version will support future
implementations of the protocol.Regards,
Dor
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
Quoted 8 lines · Luiz Capitulino
Yes, having a library was suggested by Amit some months ago. The
problem is that it has various issues wrt maintainability.For example, libvirt is able to run two instances of different
versions of qemu at the same time. How to handle this if you update libmonitor.so?
You could use dlopen(). The problem with a library is that it assumes everyone is still trapped in C, which is not the case.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
Quoted 9 lines · Avi Kivity
Quoted 8 lines · Luiz Capitulino
Yes, having a library was suggested by Amit some months ago. The
problem is that it has various issues wrt maintainability.For example, libvirt is able to run two instances of different
versions of qemu at the same time. How to handle this if you update libmonitor.so?
The sane way is to *NOT* break ABI of libmonitor.so, and not change the wire protocol in a non backwards compatible way. This is entirely doable, it just the maintainer of libmonitor/qemu to decide that ABI stability is important. So if you find an existing API / command needs to gain an extra argument, you don't change the existing API, you add a new one. Or ideally design the API upfront so that it can be extended without breaking back compatability.
Quoted 2 lines
You could use
dlopen(). The problem with a library is that it assumes everyone is still trapped in C, which is not the case.
dlopen() of a different libmonitor.so for every version of QEMU is just crazy, whether C or not. Maintain ABI & protocol back compatability with new releases and avoid the issue in the first place.
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
Quoted 13 lines · Daniel P. Berrange
Quoted 10 lines · Avi Kivity
Quoted 8 lines · Luiz Capitulino
Yes, having a library was suggested by Amit some months ago. The
problem is that it has various issues wrt maintainability.For example, libvirt is able to run two instances of different
versions of qemu at the same time. How to handle this if you update libmonitor.so?
Quoted 9 lines
The sane way is to *NOT* break ABI of libmonitor.so, and not change the wire protocol in a non backwards compatible way. This is entirely doable, it just the maintainer of libmonitor/qemu to decide that ABI stability is important. So if you find an existing API / command needs to gain an extra argument, you don't change the existing API, you add a new one. Or ideally design the API upfront so that it can be extended without breaking back compatability.
Yeah.
Quoted 8 lines
Quoted 3 lines
You could use
dlopen(). The problem with a library is that it assumes everyone is still trapped in C, which is not the case.
dlopen()of a different libmonitor.so for every version of QEMU is just crazy, whether C or not. Maintain ABI& protocol back compatability with new releases and avoid the issue in the first place.
Right, I just like dlopen().
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
In reply to Daniel P. BerrangeQuoted 11 lines · On Fri, 26 Jun 2009 10:42:24 +0100 "Daniel P. Berrange" <[email protected]>…
Quoted 9 lines · Avi Kivity
Quoted 8 lines · Luiz Capitulino
Yes, having a library was suggested by Amit some months ago. The
problem is that it has various issues wrt maintainability.For example, libvirt is able to run two instances of different
versions of qemu at the same time. How to handle this if you update libmonitor.so?
Quoted 7 lines
The sane way is to *NOT* break ABI of libmonitor.so, and not change the wire protocol in a non backwards compatible way. This is entirely doable, it just the maintainer of libmonitor/qemu to decide that ABI stability is important. So if you find an existing API / command needs to gain an extra argument, you don't change the existing API, you add a new one. Or ideally design the API upfront so that it can be extended without breaking back compatability.
[ Looks like my (evil) twin brother has taken the keyboard while I
was away and has started saying libmonitor.so non-sense. ]
I think I have mixed up problems here, but anyway, working on a C library
seems a step back to me. I believe that defining the API for it will generate more discussion than the protocol, will probably be harder to write and to document too.
In a general way the protocol idea is very simple: we just have to
make monitor's output parsable. As Anthony has put it, the patchset is not far from a mergeable state. I can't even image how long it would take to have the C API in the same shape.
That said, I think the protocol discussion went way too far. Of
course that I want to do the right thing and we should always be concerned with the future, but the priority here is not to design/choose the next-ultra-super-hiper-mathematically-proven-and-loved protocol.
So, IMHO both solutions (QMP and JSON) solves the problem and I
would work on either one. I just would like that Anthony and Avi get in agreement, because the project will fail if it becomes one more difference between qemu and qemu-kvm.
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
Quoted 5 lines · Luiz Capitulino
So, IMHO both solutions (QMP and JSON) solves the problem and I
would work on either one. I just would like that Anthony and Avi get in agreement, because the project will fail if it becomes one more difference between qemu and qemu-kvm.
There's no danger of a diverging implementation in this case since no one is proposing to have different monitor protocols. We just need to find the best protocol. Anthony's looking for minimal churn for the existing monitor command set and for libvirt, while I am considering the additional effort for new commands and for new clients. It really isn't very complicated, and the thread only got so long because the topic is relatively simple. Post an RFC and a mile-long patchset about changing TCG to SSA form, and see how you get no replies.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
Quoted 9 lines · Avi Kivity
Quoted 6 lines · Luiz Capitulino
So, IMHO both solutions (QMP and JSON) solves the problem and I
would work on either one. I just would like that Anthony and Avi get in agreement, because the project will fail if it becomes one more difference between qemu and qemu-kvm.
Quoted 8 lines
There's no danger of a diverging implementation in this case since no one
is proposing to have different monitor protocols. We just need to find the best protocol. Anthony's looking for minimal churn for the existing monitor command set and for libvirt, while I am considering the additional effort for new commands and for new clients. It really isn't very complicated, and the thread only got so long because the topic is relatively simple. Post an RFC and a mile-long patchset about changing TCG to SSA form, and see how you get no replies.
No, I think the trick is to have the mile-long patchset that twiddles something in the block layer. It can potentially cause deadly bugs for every guest and the benefit may not be obvious.
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
In reply to Avi KivityQuoted 8 lines · Avi Kivity
Quoted 6 lines · Luiz Capitulino
So, IMHO both solutions (QMP and JSON) solves the problem and I would work on either one. I just would like that Anthony and Avi get in agreement, because the project will fail if it becomes one more difference between qemu and qemu-kvm.
Quoted 5 lines
There's no danger of a diverging implementation in this case since no one is proposing to have different monitor protocols. We just need to find the best protocol. Anthony's looking for minimal churn for the existing monitor command set and for libvirt, while I am considering the additional effort for new commands and for new clients.
I'm with Avi on this issue, but I will be happy as long as the protocol is precisely described and extensible for future. Moreover I believe that converting the current code to use a new function like monitor_print_data could be done now even without knowing the exact details of the on-wire protocol. The monitor_print_data function could be then adjusted to understand the protocol specifics and emit the data accordingly.
Quoted 4 lines
It really isn't very complicated, and
the thread only got so long because the topic is relatively simple. Post an RFC and a mile-long patchset about changing TCG to SSA form, and see how you get no replies.
I wouldn't even dare to push the SSA patch... Mile-long doesn't describe it precisely enough. Imagine it was applied to all the targets.
F.
target-arm/translate.c | 3846 +++++++++++++++++++++++++++--------------------- tcg/tcg.c | 327 +++- tcg/tcg.h | 2 +- tcg/tcg-op.h | 1614 ++++++++++++-------
4 files changed, 3483 insertions(+), 2306 deletions(-)
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
Quoted 17 lines · Filip Navara
Quoted 8 lines · Avi Kivity
Quoted 6 lines · Luiz Capitulino
So, IMHO both solutions (QMP and JSON) solves the problem and I
would work on either one. I just would like that Anthony and Avi get in agreement, because the project will fail if it becomes one more difference between qemu and qemu-kvm.Quoted 6 lines
There's no danger of a diverging implementation in this case since no one is proposing to have different monitor protocols. We just need to find the best protocol. Anthony's looking for minimal churn for the existing monitor command set and for libvirt, while I am considering the additional effort for new commands and for new clients.
Quoted 8 lines
I'm with Avi on this issue, but I will be happy as long as the protocol is precisely described and extensible for future. Moreover I believe that converting the current code to use a new function like
monitor_print_datacould be done now even without knowing the exact details of the on-wire protocol. Themonitor_print_datafunction could be then adjusted to understand the protocol specifics and emit the data accordingly.
In terms of implementation, I think we could structure the human and machine monitor as follows:
- the human monitor parses a command (with the current parser) and generates a dict
- the machine monitor (using a json-like parser) parses its input which contains a parameter dict
- the dict is handed off to common code
- common code does something, returns an error code and a return value
- the human monitor prints the return value (with the current printfs)
- the machine monitor packs the return value according to the protocol and lets the json-like emitter emit it
So we don't have to duplicate parsers an emitters; instead we have to write some accessor functions for generalized values (which can come from the human monitor, command line, machine monitor, and maybe config file).
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
In reply to Filip NavaraQuoted 10 lines · Filip Navara
Quoted 4 lines · Avi Kivity
It really isn't very complicated, and
the thread only got so long because the topic is relatively simple. Post an RFC and a mile-long patchset about changing TCG to SSA form, and see how you get no replies.I wouldn't even dare to push the SSA patch... Mile-long doesn't
describe it precisely enough. Imagine it was applied to all the targets.
Is there a speed benefit from using the SSA form?
In theory, a new code generator could be added alongside the current one and when (if?) everyone is happy with it, switch to the new one.
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
Quoted 9 lines · Blue Swirl
Quoted 7 lines · Filip Navara
Quoted 4 lines · Avi Kivity
It really isn't very complicated, and
the thread only got so long because the topic is relatively simple. Post an RFC and a mile-long patchset about changing TCG to SSA form, and see how you get no replies.
Quoted 7 lines
Quoted 4 lines
I wouldn't even dare to push the SSA patch... Mile-long doesn't
describe it precisely enough. Imagine it was applied to all the targets.Is there a speed benefit from using the SSA form?
I'm no compiler expert. My understanding is that SSA enables optimizations, so translation speed would decrease, but generated code speed would increase.
Other jits increase optimization effort on long running loops, so if a code is run heavily it receives additional attention from the optimizer.
Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
In reply to Blue SwirlQuoted 8 lines · Blue Swirl
Quoted 7 lines · Filip Navara
Quoted 4 lines · Avi Kivity
It really isn't very complicated, and
the thread only got so long because the topic is relatively simple. Post an RFC and a mile-long patchset about changing TCG to SSA form, and see how you get no replies.
Quoted 5 lines
Quoted 3 lines
I wouldn't even dare to push the SSA patch... Mile-long doesn't
describe it precisely enough. Imagine it was applied to all the targets.Is there a speed benefit from using the SSA form?
You can do some nice optimisations on SSA form, but you can do them without SSA too. It's just that SSA makes it easier to think about some things.
SSA is only useful for interesting control flow graphs with many basic blocks, though. As far as I know, QEMU only translates code in linear instruction traces - so most SSA optimisations don't apply, and those which remain are easy to think about in any representation.
-- Jamie
Re: [Qemu-devel] Re: [PATCH 08/11] QMP: Port balloon command
In reply to Dor LaorQuoted 7 lines · Dor Laor
Quoted 5 lines
3) Get the structured printfs in order.
4) MergeI won't want QMP to be exposed as a usable interface until all commands are converted. This means holding off on the last patch I think. I don't think we'll get QMP for 0.11. I think it's likely going to be a 0.12 feature. However, I'd like to start merging this series as soon as humanly possible. I think it's already pretty close.
If we push 'monitor standardization' into 0.11, the version will support future implementations of the protocol.
We feature freeze in 2 weeks. There's no way we can have it ready by then.
Regards,
Anthony Liguori
Quoted 2 lines
Regards,
Dor
PATCH 9/11 QMP: Port 'info blockstats' command
In reply to a message not in the archive--- block.c | 18 +++++++++--------- monitor.c | 3 ++- 2 files changed, 11 insertions(+), 10 deletions(-)
block.c +9 -9
@@ -1102,15 +1102,15 @@ void bdrv_info_stats(Monitor *mon)BlockDriverState *bs;for (bs = bdrv_first; bs !=NULL ; bs = bs->next) { -monitor_printf (mon,"%s:" -" rd_bytes=%" PRIu64 -" wr_bytes=%" PRIu64 -" rd_operations=%" PRIu64 -" wr_operations=%" PRIu64 -"\n" , - bs->device_name, - bs->rd_bytes, bs->wr_bytes, - bs->rd_ops, bs->wr_ops); +monitor_printf_data (mon,"%s:" +" rd_bytes=%" PRIu64 +" wr_bytes=%" PRIu64 +" rd_operations=%" PRIu64 +" wr_operations=%" PRIu64 +"\n" , + bs->device_name, + bs->rd_bytes, bs->wr_bytes, + bs->rd_ops, bs->wr_ops); } }
monitor.c +2 -1
@@ -2512,7 +2512,8 @@ static int valid_control_cmd(Monitor *mon, const char *cmd,"balloon" , NULL };const char *valid_infos[] = {"balloon" , -NULL }; +"blockstats" , +NULL };if (!monitor_ctrl_mode (mon)) {/* all commands are valid in user-mode mode */
Signature · 1 line
1.6.3.GIT
[Qemu-devel] Re: [PATCH 09/11] QMP: Port 'info blockstats' command
Quoted 23 lines · Luiz Capitulino
Signed-off-by Luiz Capitulino<[email protected]>--- block.c | 18 +++++++++--------- monitor.c | 3 ++- 2 files changed, 11 insertions(+), 10 deletions(-)block.c +0
-9@@ -1102,15 +1102,15 @@ void bdrv_info_stats(Monitor *mon)BlockDriverState *bs;for (bs = bdrv_first; bs !=NULL ; bs = bs->next) { -monitor_printf (mon,"%s:" -" rd_bytes=%" PRIu64 -" wr_bytes=%" PRIu64 -" rd_operations=%" PRIu64 -" wr_operations=%" PRIu64 -"\n" , - bs->device_name, - bs->rd_bytes, bs->wr_bytes, - bs->rd_ops, bs->wr_ops);
block.c void bdrv_info_stats(Monitor *mon) · line 1114
Quoted 10 lines
block.c +9
-0+monitor_printf_data (mon,"%s:" +" rd_bytes=%" PRIu64 +" wr_bytes=%" PRIu64 +" rd_operations=%" PRIu64 +" wr_operations=%" PRIu64 +"\n" , + bs->device_name, + bs->rd_bytes, bs->wr_bytes, + bs->rd_ops, bs->wr_ops);
It may be easier for control mode to drop the variable= prefixes (they are described in the spec anyway).
Signature · 1 line
error compiling committee.c: too many arguments to function
[Qemu-devel] Re: [PATCH 09/11] QMP: Port 'info blockstats' command
Quoted 29 lines · Avi Kivity
Quoted 28 lines · Luiz Capitulino
Signed-off-by Luiz Capitulino<[email protected]>--- block.c | 18 +++++++++--------- monitor.c | 3 ++- 2 files changed, 11 insertions(+), 10 deletions(-)block.c +5
-9@@ -1102,15 +1102,15 @@ void bdrv_info_stats(Monitor *mon)BlockDriverState *bs;for (bs = bdrv_first; bs !=NULL ; bs = bs->next) { -monitor_printf (mon,"%s:" -" rd_bytes=%" PRIu64 -" wr_bytes=%" PRIu64 -" rd_operations=%" PRIu64 -" wr_operations=%" PRIu64 -"\n" , - bs->device_name, - bs->rd_bytes, bs->wr_bytes, - bs->rd_ops, bs->wr_ops); +monitor_printf_data (mon,"%s:" +" rd_bytes=%" PRIu64 +" wr_bytes=%" PRIu64 +" rd_operations=%" PRIu64 + " wr_operations=%" PRIu64
Quoted 9 lines
block.c void bdrv_info_stats(Monitor *mon) · line 1116
Quoted 5 lines
block.c +4
-0+"\n" , + bs->device_name, + bs->rd_bytes, bs->wr_bytes, + bs->rd_ops, bs->wr_ops);It may be easier for control mode to drop the variable= prefixes (they are described in the spec anyway).
Isn't monitor_printf_data also used for pretty-printing in case of human-operated monitors? I think we need to keep this - or provide two output formats which are filtered by the monitor based on the the target terminal's mode. But that sounds like a lot of work /wrt conversion and long-term maintenance.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
[Qemu-devel] Re: [PATCH 09/11] QMP: Port 'info blockstats' command
Quoted 31 lines · Jan Kiszka
Quoted 20 lines
Quoted 20 lines
block.c +5
-9@@ -1102,15 +1102,15 @@ void bdrv_info_stats(Monitor *mon)BlockDriverState *bs;for (bs = bdrv_first; bs !=NULL ; bs = bs->next) { -monitor_printf (mon,"%s:" -" rd_bytes=%" PRIu64 -" wr_bytes=%" PRIu64 -" rd_operations=%" PRIu64 -" wr_operations=%" PRIu64 -"\n" , - bs->device_name, - bs->rd_bytes, bs->wr_bytes, - bs->rd_ops, bs->wr_ops); +monitor_printf_data (mon,"%s:" +" rd_bytes=%" PRIu64 +" wr_bytes=%" PRIu64 +" rd_operations=%" PRIu64 + " wr_operations=%" PRIu64Quoted 10 lines
block.c void bdrv_info_stats(Monitor *mon)
Quoted 6 lines
block.c +4
-0+"\n" , + bs->device_name, + bs->rd_bytes, bs->wr_bytes, + bs->rd_ops, bs->wr_ops);It may be easier for control mode to drop the variable= prefixes (they are described in the spec anyway).
Quoted 2 lines
Isn't
monitor_printf_dataalso used for pretty-printing in case of human-operated monitors?
It is.
Quoted 5 lines
I think we need to keep this - or provide two
output formats which are filtered by the monitor based on the the target terminal's mode. But that sounds like a lot of work /wrt conversion and long-term maintenance.
I prefer separate printfs. We don't want to allow choices made for the human protocol to influence the machine protocol, the whole point of the machine protocol was to have a clean break.
Signature · 1 line
error compiling committee.c: too many arguments to function
[Qemu-devel] Re: [PATCH 09/11] QMP: Port 'info blockstats' command
Quoted 43 lines · Avi Kivity
Quoted 31 lines · Jan Kiszka
Quoted 20 lines
Quoted 20 lines
block.c +5
-9@@ -1102,15 +1102,15 @@ void bdrv_info_stats(Monitor *mon)BlockDriverState *bs;for (bs = bdrv_first; bs !=NULL ; bs = bs->next) { -monitor_printf (mon,"%s:" -" rd_bytes=%" PRIu64 -" wr_bytes=%" PRIu64 -" rd_operations=%" PRIu64 -" wr_operations=%" PRIu64 -"\n" , - bs->device_name, - bs->rd_bytes, bs->wr_bytes, - bs->rd_ops, bs->wr_ops); +monitor_printf_data (mon,"%s:" +" rd_bytes=%" PRIu64 +" wr_bytes=%" PRIu64 +" rd_operations=%" PRIu64 + " wr_operations=%" PRIu64Quoted 10 lines
block.c void bdrv_info_stats(Monitor *mon)
Quoted 6 lines
block.c +4
-0+"\n" , + bs->device_name, + bs->rd_bytes, bs->wr_bytes, + bs->rd_ops, bs->wr_ops);It may be easier for control mode to drop the variable= prefixes (they are described in the spec anyway).
Quoted 2 lines
Isn't
monitor_printf_dataalso used for pretty-printing in case of human-operated monitors?It is.
Quoted 5 lines
I think we need to keep this - or provide two
output formats which are filtered by the monitor based on the the target terminal's mode. But that sounds like a lot of work /wrt conversion and long-term maintenance.
Quoted 3 lines
I prefer separate printfs. We don't want to allow choices made for the human protocol to influence the machine protocol, the whole point of the machine protocol was to have a clean break.
OK... yeah, makes sense. Then the above hunk must not remove the existing monitor_printf, just add new monitor_printf_data.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Re: [Qemu-devel] [PATCH 09/11] QMP: Port 'info blockstats' command
In reply to Luiz CapitulinoQuoted 23 lines · Luiz Capitulino
Signed-off-by Luiz Capitulino <[email protected]>--- block.c | 18 +++++++++--------- monitor.c | 3 ++- 2 files changed, 11 insertions(+), 10 deletions(-)block.c +0
-9@@ -1102,15 +1102,15 @@ void bdrv_info_stats(Monitor *mon)BlockDriverState *bs;for (bs = bdrv_first; bs !=NULL ; bs = bs->next) { -monitor_printf (mon,"%s:" -" rd_bytes=%" PRIu64 -" wr_bytes=%" PRIu64 -" rd_operations=%" PRIu64 -" wr_operations=%" PRIu64 -"\n" , - bs->device_name, - bs->rd_bytes, bs->wr_bytes, - bs->rd_ops, bs->wr_ops);
block.c void bdrv_info_stats(Monitor *mon) · line 1114
Quoted 10 lines
block.c +9
-0+monitor_printf_data (mon,"%s:" +" rd_bytes=%" PRIu64 +" wr_bytes=%" PRIu64 +" rd_operations=%" PRIu64 +" wr_operations=%" PRIu64 +"\n" , + bs->device_name, + bs->rd_bytes, bs->wr_bytes, + bs->rd_ops, bs->wr_ops);
monitor_printf_list(mon, bs->device_name,
"rd_bytes", MON_BYTES, bs->rd_bytes,
"wr_bytes", MON_BYTES, bs->wr_bytes,
"rd_operations", MON_BYTES, bs->rd_ops,
"wr_operations", MON_BYTES, bs->wr_ops,
NULL);Regards,
Anthony Liguori
PATCH 10/11 QMP: Introduce basic events
In reply to a message not in the archiveReboot, shutdown and powerdown are very basic events and can be added together.
--- monitor.c | 9 +++++++++ monitor.h | 3 +++ vl.c | 3 +++ 3 files changed, 15 insertions(+), 0 deletions(-)
monitor.c +9 -0
@@ -276,6 +276,15 @@ void monitor_notify_event(MonitorEvent event)monitor_puts (cur_mon,"* EVENT " );switch (event) { +case EVENT_REBOOT : +monitor_puts (cur_mon,"reboot" ); +break ; +case EVENT_SHUTDOWN : +monitor_puts (cur_mon,"shutdown" ); +break ; +case EVENT_POWERDOWN : +monitor_puts (cur_mon,"powerdown" ); +break ;case EVENT_MAX :// Avoid gcc warning, will never get here break ;
monitor.h +3 -0
@@ -14,6 +14,9 @@ extern Monitor *cur_mon;/* QMP events */ typedef enum MonitorEvent { +EVENT_REBOOT , +EVENT_SHUTDOWN , +EVENT_POWERDOWN ,EVENT_MAX , }MonitorEvent ;
vl.c +3 -0
@@ -3693,18 +3693,21 @@ void qemu_system_reset_request(void) reset_requested =1 ; }qemu_notify_event (); +monitor_notify_event (EVENT_REBOOT ); }void qemu_system_shutdown_request (void ) { shutdown_requested =1 ;qemu_notify_event (); +monitor_notify_event (EVENT_SHUTDOWN ); }void qemu_system_powerdown_request (void ) { powerdown_requested =1 ;qemu_notify_event (); +monitor_notify_event (EVENT_POWERDOWN ); }#ifdef CONFIG_IOTHREAD
Signature · 1 line
1.6.3.GIT
[Qemu-devel] Re: [PATCH 10/11] QMP: Introduce basic events
Quoted 4 lines · Luiz Capitulino
Reboot, shutdown and powerdown are very basic events and can be added together.
Reboot only exists within a guest. qemu sees a reset (there are reboots not associated with a reset, for example kexec).
Quoted 16 lines
vl.c +1
-0@@ -3693,18 +3693,21 @@ void qemu_system_reset_request(void) reset_requested =1 ; }qemu_notify_event (); +monitor_notify_event (EVENT_REBOOT ); }void qemu_system_shutdown_request (void ) { shutdown_requested =1 ;qemu_notify_event ();
vl.c void qemu_system_reset_request(void) · line 3712
Quoted 10 lines
vl.c +2
-0+monitor_notify_event (EVENT_SHUTDOWN ); }void qemu_system_powerdown_request (void ) { powerdown_requested =1 ;qemu_notify_event (); +monitor_notify_event (EVENT_POWERDOWN ); }
Maybe it's better to signal the event when the reset etc. actually takes place, to avoid subtle races.
Signature · 1 line
error compiling committee.c: too many arguments to function
[Qemu-devel] Re: [PATCH 10/11] QMP: Introduce basic events
Quoted 8 lines · On Tue, 23 Jun 2009 12:46:06 +0300 Avi Kivity <[email protected]> wrote:
Quoted 4 lines · Luiz Capitulino
Reboot, shutdown and powerdown are very basic events and can be added together.
Reboot only exists within a guest. qemu sees a reset (there are reboots not associated with a reset, for example kexec).
Yeah, will fix.
Quoted 18 lines
Quoted 18 lines
vl.c +2
-0@@ -3693,18 +3693,21 @@ void qemu_system_reset_request(void) reset_requested =1 ; }qemu_notify_event (); +monitor_notify_event (EVENT_REBOOT ); }void qemu_system_shutdown_request (void ) { shutdown_requested =1 ;qemu_notify_event (); +monitor_notify_event (EVENT_SHUTDOWN ); }
Quoted 10 lines
vl.c void qemu_system_reset_request(void) · line 3712
Quoted 7 lines
vl.c +1
-0void qemu_system_powerdown_request (void ) { powerdown_requested =1 ;qemu_notify_event (); +monitor_notify_event (EVENT_POWERDOWN ); }Maybe it's better to signal the event when the reset etc. actually takes place, to avoid subtle races.
This concern was raised by Daniel as well, will change
this too.
[Qemu-devel] Re: [PATCH 10/11] QMP: Introduce basic events
In reply to Luiz CapitulinoQuoted 58 lines · Luiz Capitulino
Reboot, shutdown and powerdown are very basic events and can be added together.
Signed-off-by Luiz Capitulino <[email protected]>--- monitor.c | 9 +++++++++ monitor.h | 3 +++ vl.c | 3 +++ 3 files changed, 15 insertions(+), 0 deletions(-)monitor.c +9
-0@@ -276,6 +276,15 @@ void monitor_notify_event(MonitorEvent event)monitor_puts (cur_mon,"* EVENT " );switch (event) { +case EVENT_REBOOT : +monitor_puts (cur_mon,"reboot" ); +break ; +case EVENT_SHUTDOWN : +monitor_puts (cur_mon,"shutdown" ); +break ; +case EVENT_POWERDOWN : +monitor_puts (cur_mon,"powerdown" ); +break ;case EVENT_MAX :// Avoid gcc warning, will never get here break ;monitor.h +3
-0@@ -14,6 +14,9 @@ extern Monitor *cur_mon;/* QMP events */ typedef enum MonitorEvent { +EVENT_REBOOT , +EVENT_SHUTDOWN , +EVENT_POWERDOWN ,EVENT_MAX , }MonitorEvent ;vl.c +1
-0@@ -3693,18 +3693,21 @@ void qemu_system_reset_request(void) reset_requested =1 ; }qemu_notify_event (); +monitor_notify_event (EVENT_REBOOT ); }void qemu_system_shutdown_request (void ) { shutdown_requested =1 ;
vl.c void qemu_system_reset_request(void) · line 3711
Quoted 10 lines
vl.c +2
-0qemu_notify_event (); +monitor_notify_event (EVENT_SHUTDOWN ); }void qemu_system_powerdown_request (void ) { powerdown_requested =1 ;qemu_notify_event (); +monitor_notify_event (EVENT_POWERDOWN ); }
I could be mis-reading this code, but this implies that the mgmt app would get these events at the time the action is requested, but not neccessarily actually acted upon. Wouldn't you want to be receiving the events immediately after the action has been completed ?
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
PATCH 11/11 QMP: Command-line flag to enable control mode
In reply to a message not in the archiveThis change adds a flag called 'control' to the already existing '-monitor' command-line option. This flag can be used to enable control mode.
Its syntax is:
qemu [...] -monitor control,<device>
Where <device> is a chardev (excluding 'vc', for obvious reasons).
For example:
$ qemu [...] -monitor control,tcp:localhost:4444,server
Will run QEMU in control mode, waiting for a client TCP connection on localhost port 4444.
--- monitor.c | 19 +++++++++++++++++++ monitor.h | 1 + vl.c | 5 +++-- 3 files changed, 23 insertions(+), 2 deletions(-)
monitor.c +19 -0
@@ -3192,6 +3192,25 @@ static void monitor_event(void *opaque, int event)* End: */ +const char *monitor_cmdline_parse (const char *cmdline,int *flags) +{ +const char *dev; + + *flags =MONITOR_IS_DEFAULT ; +if (strstart (cmdline,"control," , &dev)) { +if (strstart (dev,"vc" ,NULL )) { +fprintf (stderr,"qemu: control mode is for low-level interaction " ); +fprintf (stderr,"cannot be used with device 'vc'\n" ); +exit (1 ); + } + *flags |=MONITOR_USE_CONTROL ; +return dev; + } + + *flags |=MONITOR_USE_READLINE ; +return cmdline; +} +void monitor_init (CharDriverState *chr,int flags) {static int is_first_init =1 ;
monitor.h +1 -0
@@ -20,6 +20,7 @@ typedef enum MonitorEvent {EVENT_MAX , } MonitorEvent; +const char *monitor_cmdline_parse (const char *cmdline,int *flags);void monitor_init (CharDriverState *chr,int flags);int monitor_suspend (Monitor *mon);
vl.c +3 -2
@@ -4977,6 +4977,7 @@ int main(int argc, char **argv, char **envp)const char *r, *optarg;CharDriverState *monitor_hd =NULL ;const char *monitor_device; +int monitor_flags =MONITOR_IS_DEFAULT |MONITOR_USE_READLINE ;const char *serial_devices[MAX_SERIAL_PORTS ];int serial_device_index;const char *parallel_devices[MAX_PARALLEL_PORTS ];
@@ -5462,7 +5463,7 @@ int main(int argc, char **argv, char **envp)break ; }case QEMU_OPTION_monitor: - monitor_device = optarg; + monitor_device =monitor_cmdline_parse (optarg, &monitor_flags);break ;case QEMU_OPTION_serial:if (serial_device_index >=MAX_SERIAL_PORTS ) {
@@ -6150,7 +6151,7 @@ int main(int argc, char **argv, char **envp)qemu_chr_initial_reset ();if (monitor_device && monitor_hd) -monitor_init (monitor_hd,MONITOR_USE_READLINE |MONITOR_IS_DEFAULT ); +monitor_init (monitor_hd, monitor_flags);for (i =0 ; i <MAX_SERIAL_PORTS ; i++) {const char *devname = serial_devices[i];
Signature · 1 line
1.6.3.GIT
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 10 lines · Luiz Capitulino
This change adds a flag called 'control' to the already existing '-monitor' command-line option. This flag can be used to enable control mode.
Its syntax is:
qemu [...] -monitor control,<device>
Where <device> is a chardev (excluding 'vc', for obvious reasons).
Quoted 8 lines
For example:
$ qemu [...] -monitor control,tcp:localhost:4444,serverWill run QEMU in control mode, waiting for a client TCP connection on localhost port 4444.
Signed-off-by Luiz Capitulino <[email protected]>
At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor instances". I think Anthony posted such a patch before. Now we should really include this as your extension may block the stand-alone -monitor switch for the control channel, preventing to additionally set up one (or more) for debugging purposes.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 18 lines · Jan Kiszka
Quoted 17 lines · Luiz Capitulino
This change adds a flag called 'control' to the already existing '-monitor' command-line option. This flag can be used to enable control mode.
Its syntax is:
qemu [...] -monitor control,<device>
Where <device> is a chardev (excluding 'vc', for obvious reasons).
For example:
$ qemu [...] -monitor control,tcp:localhost:4444,serverWill run QEMU in control mode, waiting for a client TCP connection on localhost port 4444.
Quoted 7 lines
Quoted 1 line
Signed-off-by Luiz Capitulino <[email protected]>At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor instances". I think Anthony posted such a patch before. Now we should really include this as your extension may block the stand-alone -monitor switch for the control channel, preventing to additionally set up one (or more) for debugging purposes.
If we support multiple monitors, it might be desirable to allow the app owning the main 'control' monitor channel to be able to indicate that additional monitor channels are read-only. eg, so libvirt could allow the user to connect to the monitor to run 'info' commands for debug support without risk of having state changed behind its back
Regards,
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 26 lines · Daniel P. Berrange
Quoted 18 lines · Jan Kiszka
Quoted 17 lines · Luiz Capitulino
This change adds a flag called 'control' to the already existing '-monitor' command-line option. This flag can be used to enable control mode.
Its syntax is:
qemu [...] -monitor control,<device>
Where <device> is a chardev (excluding 'vc', for obvious reasons).
For example:
$ qemu [...] -monitor control,tcp:localhost:4444,serverWill run QEMU in control mode, waiting for a client TCP connection on localhost port 4444.
Quoted 6 lines
Quoted 1 line
Signed-off-by Luiz Capitulino <[email protected]>At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor instances". I think Anthony posted such a patch before. Now we should really include this as your extension may block the stand-alone -monitor switch for the control channel, preventing to additionally set up one (or more) for debugging purposes.
Quoted 5 lines
If we support multiple monitors, it might be desirable to allow the app owning the main 'control' monitor channel to be able to indicate that additional monitor channels are read-only. eg, so libvirt could allow the user to connect to the monitor to run 'info' commands for debug support without risk of having state changed behind its back
Couldn't libvirt deal with update given we provide them as events? Otherwise, your suggestion makes sense, definitely as long as it would wreck libvirt's internal house keeping or for commands that are not synchronizable.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 7 lines · Jan Kiszka
Quoted 6 lines
If we support multiple monitors, it might be desirable to allow the app owning the main 'control' monitor channel to be able to indicate that additional monitor channels are read-only. eg, so libvirt could allow the user to connect to the monitor to run 'info' commands for debug support without risk of having state changed behind its back
Quoted 5 lines
Couldn't libvirt deal with update given we provide them as events? Otherwise, your suggestion makes sense, definitely as long as it would wreck libvirt's internal house keeping or for commands that are not synchronizable.
It would be a nightmare, consider both libvirt and a user hotplugging something into the same pci slot, or a user starting migration, or quitting, or ...
Having -monitor readonly makes sense.
Signature · 1 line
error compiling committee.c: too many arguments to function
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 14 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 6 lines
If we support multiple monitors, it might be desirable to allow the app owning the main 'control' monitor channel to be able to indicate that additional monitor channels are read-only. eg, so libvirt could allow the user to connect to the monitor to run 'info' commands for debug support without risk of having state changed behind its back
Quoted 5 lines
Couldn't libvirt deal with update given we provide them as events? Otherwise, your suggestion makes sense, definitely as long as it would wreck libvirt's internal house keeping or for commands that are not synchronizable.
Quoted 3 lines
It would be a nightmare, consider both libvirt and a user hotplugging something into the same pci slot, or a user starting migration, or quitting, or ...
Migration, yes, but hot-plugging is resolvable - given config update events.
Quoted 2 lines
Having -monitor readonly makes sense.
But not for all features, only those libvirt is capable to handle (I think there are quite a few qemu specifics libvirt does not bother about) and only as long as there is no proper synchronization. Again, migration and save/restore will continue to require exclusive access, but the rest is just a question of proper synchronization IMHO.
See, I don't want to kill my management app just because I attached to a guest via gdb and start injecting reconfiguration events for testing purposes (e.g. attach/detach a USB device for which I develop a driver).
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 7 lines · Jan Kiszka
Quoted 4 lines
It would be a nightmare, consider both libvirt and a user hotplugging something into the same pci slot, or a user starting migration, or quitting, or ...
Migration, yes, but hot-plugging is resolvable - given config update events.
I'm not saying it's impossible, just that we don't want to do it.
Quoted 9 lines
Quoted 2 lines
Having -monitor readonly makes sense.
But not for all features, only those libvirt is capable to handle (I think there are quite a few qemu specifics libvirt does not bother about) and only as long as there is no proper synchronization. Again, migration and save/restore will continue to require exclusive access, but the rest is just a question of proper synchronization IMHO.
If libvirt "doesn't bother" about something useful, that's a libvirt bug. It's wierd to have something controlled by two parallel management channels.
Quoted 4 lines
See, I don't want to kill my management app just because I attached to a guest via gdb and start injecting reconfiguration events for testing purposes (e.g. attach/detach a USB device for which I develop a driver).
You're talking about a usb developer, not a qemu developer, working in some virtual lab environment? In that case, libvirt and the management app should certainly support usb. A random usb developer shouldn't need to ever talk to the qemu monitor.
Signature · 1 line
error compiling committee.c: too many arguments to function
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 11 lines · Avi Kivity
Quoted 8 lines · Jan Kiszka
Quoted 4 lines
It would be a nightmare, consider both libvirt and a user hotplugging something into the same pci slot, or a user starting migration, or quitting, or ...
Migration, yes, but hot-plugging is resolvable - given config update events.
I'm not saying it's impossible, just that we don't want to do it.
I agree it's not something that is mandatory for the first versions, but disagree that we should not strive for achieving this level of support mid- to long-term. No design decision made today should prevent it, but rather keep that door open. Also for robustness reasons (if you loose contact to your qemu backend and need to resync the management frontend).
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
Having -monitor readonly makes sense.
But not for all features, only those libvirt is capable to handle (I think there are quite a few qemu specifics libvirt does not bother about) and only as long as there is no proper synchronization. Again, migration and save/restore will continue to require exclusive access, but the rest is just a question of proper synchronization IMHO.
Quoted 3 lines
If libvirt "doesn't bother" about something useful, that's a libvirt bug. It's wierd to have something controlled by two parallel management channels.
I think the libvirt policy is that things which are not generic enough to be supported by >1 hypervisor are left alone.
Quoted 10 lines
Quoted 5 lines
See, I don't want to kill my management app just because I attached to a guest via gdb and start injecting reconfiguration events for testing purposes (e.g. attach/detach a USB device for which I develop a driver).
You're talking about a usb developer, not a qemu developer, working in some virtual lab environment? In that case, libvirt and the management app should certainly support usb. A random usb developer shouldn't need to ever talk to the qemu monitor.
I'm talking about, just as one example, sitting in front of my gdb [frontend], doing guest kernel [driver] debugging and issuing "monitor whatever" commands from one place, ie. not having to switch between management app and debugging interface back and forth.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 9 lines · Jan Kiszka
Quoted 2 lines
I'm not saying it's impossible, just that we don't want to do it.
I agree it's not something that is mandatory for the first versions, but disagree that we should not strive for achieving this level of support mid- to long-term. No design decision made today should prevent it, but rather keep that door open. Also for robustness reasons (if you loose contact to your qemu backend and need to resync the management frontend).
No objections here. Certainly reconnect is important.
Quoted 8 lines
Quoted 4 lines
If libvirt "doesn't bother" about something useful, that's a libvirt bug. It's wierd to have something controlled by two parallel management channels.
I think the libvirt policy is that things which are not generic enough to be supported by>1 hypervisor are left alone.
That's a bit offtopic here, but it appears to me this dooms libvirt to be useless in all but the most basic use cases. Hypervisor writers try to new features to differentiate their products, and forcing users off libvirt in order to use them seems to be counterproductive.
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
See, I don't want to kill my management app just because I attached to a guest via gdb and start injecting reconfiguration events for testing purposes (e.g. attach/detach a USB device for which I develop a driver).
You're talking about a usb developer, not a qemu developer, working in some virtual lab environment? In that case, libvirt and the management app should certainly support usb. A random usb developer shouldn't need to ever talk to the qemu monitor.
Quoted 5 lines
I'm talking about, just as one example, sitting in front of my gdb [frontend], doing guest kernel [driver] debugging and issuing "monitor whatever" commands from one place, ie. not having to switch between management app and debugging interface back and forth.
Can gdb issue monitor commands?
Signature · 1 line
error compiling committee.c: too many arguments to function
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 17 lines · Avi Kivity
Quoted 9 lines · Jan Kiszka
Quoted 2 lines
I'm not saying it's impossible, just that we don't want to do it.
I agree it's not something that is mandatory for the first versions, but disagree that we should not strive for achieving this level of support mid- to long-term. No design decision made today should prevent it, but rather keep that door open. Also for robustness reasons (if you loose contact to your qemu backend and need to resync the management frontend).
No objections here. Certainly reconnect is important.
Quoted 5 lines
Quoted 4 lines
If libvirt "doesn't bother" about something useful, that's a libvirt bug. It's wierd to have something controlled by two parallel management channels.
Quoted 8 lines
Quoted 3 lines
I think the libvirt policy is that things which are not generic enough to be supported by>1 hypervisor are left alone.
That's a bit offtopic here, but it appears to me this dooms libvirt to be useless in all but the most basic use cases. Hypervisor writers try to new features to differentiate their products, and forcing users off libvirt in order to use them seems to be counterproductive.
That was my personal interpretation, maybe based on incomplete information.
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 8 lines
See, I don't want to kill my management app just because I attached to a
guest via gdb and start injecting reconfiguration events for testing purposes (e.g. attach/detach a USB device for which I develop a driver).You're talking about a usb developer, not a qemu developer, working in some virtual lab environment? In that case, libvirt and the management app should certainly support usb. A random usb developer shouldn't need to ever talk to the qemu monitor.
Quoted 7 lines
Quoted 5 lines
I'm talking about, just as one example, sitting in front of my gdb [frontend], doing guest kernel [driver] debugging and issuing "monitor whatever" commands from one place, ie. not having to switch between management app and debugging interface back and forth.
Can gdb issue monitor commands?
Yes.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 11 lines · Jan Kiszka
Quoted 8 lines
Quoted 6 lines
I'm talking about, just as one example, sitting in front of my gdb [frontend], doing guest kernel [driver] debugging and issuing "monitor whatever" commands from one place, ie. not having to switch between management app and debugging interface back and forth.
Can gdb issue monitor commands?
Yes.
How does one do that? seems a very useful feature.
I could say that management can proxy the gdb packets and thus support its own cli (in fact, it must if it wants to support live migration), but that's really an edge case and I doubt anyone will do that, so you have a good point.
Signature · 1 line
error compiling committee.c: too many arguments to function
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 7 lines · Avi Kivity
Quoted 6 lines · Jan Kiszka
Quoted 6 lines
Quoted 6 lines
I'm talking about, just as one example, sitting in front of my gdb [frontend], doing guest kernel [driver] debugging and issuing "monitor whatever" commands from one place, ie. not having to switch between management app and debugging interface back and forth.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Can gdb issue monitor commands?
Yes.
How does one do that? seems a very useful feature.
By spawning an additional monitor terminal, but switching off its readline support (as that is handled by gdb).
In case there is still someone out there not being aware of this ;) : we _do_ have support for multiple monitors in qemu already. Just try "-monitor X -serial mon:Y" with X!=Y.
Quoted 5 lines
I could say that management can proxy the gdb packets and thus support its own cli (in fact, it must if it wants to support live migration), but that's really an edge case and I doubt anyone will do that, so you have a good point.
Proxying would mean interpreting all "classic" monitor commands to catch those that may interfere with the mgmt app state. I also don't think that is worth the effort.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 34 lines · Jan Kiszka
Quoted 9 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 7 lines
Quoted 7 lines
I'm talking about, just as one example, sitting in front of my gdb [frontend], doing guest kernel [driver] debugging and issuing "monitor whatever" commands from one place, ie. not having to switch between management app and debugging interface back and forth.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Can gdb issue monitor commands?
Yes.
How does one do that? seems a very useful feature.
By spawning an additional monitor terminal, but switching off its readline support (as that is handled by gdb).
In case there is still someone out there not being aware of this ;) : we _do_ have support for multiple monitors in qemu already. Just try "-monitor X -serial mon:Y" with X!=Y.
Quoted 5 lines
I could say that management can proxy the gdb packets and thus support its own cli (in fact, it must if it wants to support live migration), but that's really an edge case and I doubt anyone will do that, so you have a good point.
Quoted 4 lines
Proxying would mean interpreting all "classic" monitor commands to catch those that may interfere with the mgmt app state. I also don't think that is worth the effort.
No, you would use the management system's cli which naturally enforces all of a user's permissions and is integrated with the GUI:
gui ----\
cli ----+---[ management server ] ---- [ libvirt ] ---- qemu gdb ----/
That's the only setup that can support migration without dropping sessions. That doesn't change the "I don't think it's worth the effort" state though.
Signature · 1 line
error compiling committee.c: too many arguments to function
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
In reply to Jan KiszkaQuoted 12 lines · Jan Kiszka
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Can gdb issue monitor commands?
Yes.
How does one do that? seems a very useful feature.
By spawning an additional monitor terminal, but switching off its readline support (as that is handled by gdb).
Within gdb, how do you tell it to send a line to the monitor?
Signature · 1 line
error compiling committee.c: too many arguments to function
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
In reply to Avi KivityQuoted 6 lines · Avi Kivity
Quoted 5 lines · Jan Kiszka
Quoted 4 lines
If libvirt "doesn't bother" about something useful, that's a libvirt bug. It's wierd to have something controlled by two parallel management channels.
Quoted 8 lines
Quoted 3 lines
I think the libvirt policy is that things which are not generic enough to be supported by>1 hypervisor are left alone.
That's a bit offtopic here, but it appears to me this dooms libvirt to be useless in all but the most basic use cases. Hypervisor writers try to new features to differentiate their products, and forcing users off libvirt in order to use them seems to be counterproductive.
libvirt is *not* restricted to 'lowest common denominator'. We are fine with supporting features that are only supported in 1 hypervisor. Our goal is to specify a generic representation of the feature, rather than directly exposing the underlying syntax of the HV.
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
In reply to Avi KivityQuoted 22 lines · Avi Kivity
Quoted 8 lines · Jan Kiszka
Quoted 4 lines
It would be a nightmare, consider both libvirt and a user hotplugging something into the same pci slot, or a user starting migration, or quitting, or ...
Migration, yes, but hot-plugging is resolvable - given config update events.
I'm not saying it's impossible, just that we don't want to do it.
Quoted 9 lines
Quoted 2 lines
Having -monitor readonly makes sense.
But not for all features, only those libvirt is capable to handle (I think there are quite a few qemu specifics libvirt does not bother about) and only as long as there is no proper synchronization. Again, migration and save/restore will continue to require exclusive access, but the rest is just a question of proper synchronization IMHO.
Quoted 3 lines
If libvirt "doesn't bother" about something useful, that's a libvirt bug. It's wierd to have something controlled by two parallel management channels.
It is not neccessaryily a 'bug'. The issue is that there is always a potential time lag between a feature appearing in QEMU, and it being fully supported in libvirt. So it is not a case of "doesnt' bother", but rather 'not yet known about'. I think we should make it possible for apps to be robust in this scenario, by allowing them to lockdown 2ndary monitor channels readonly.
Regards,
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Re: [Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 12 lines · Daniel P. Berrange
Quoted 11 lines
Quoted 7 lines
But not for all features, only those libvirt is capable to handle (I think there are quite a few qemu specifics libvirt does not bother about) and only as long as there is no proper synchronization. Again, migration and save/restore will continue to require exclusive access, but the rest is just a question of proper synchronization IMHO.
If libvirt "doesn't bother" about something useful, that's a libvirt bug. It's wierd to have something controlled by two parallel management channels.
Quoted 7 lines
It is not neccessaryily a 'bug'. The issue is that there is always a potential time lag between a feature appearing in QEMU, and it being fully supported in libvirt. So it is not a case of "doesnt' bother", but rather 'not yet known about'. I think we should make it possible for apps to be robust in this scenario, by allowing them to lockdown 2ndary monitor channels readonly.
In this case I don't see much of an issue, since the time to implement a feature in libvirt is probably much shorter than the time needed to implement a proper GUI, database support, etc. that's needed for most features. Short cutting through another interface is likely to cost more time than it saves.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
In reply to Avi KivityQuoted 14 lines · Avi Kivity
Quoted 7 lines · Jan Kiszka
Quoted 6 lines
If we support multiple monitors, it might be desirable to allow the app owning the main 'control' monitor channel to be able to indicate that additional monitor channels are read-only. eg, so libvirt could allow the user to connect to the monitor to run 'info' commands for debug support without risk of having state changed behind its back
Quoted 5 lines
Couldn't libvirt deal with update given we provide them as events? Otherwise, your suggestion makes sense, definitely as long as it would wreck libvirt's internal house keeping or for commands that are not synchronizable.
Quoted 3 lines
It would be a nightmare, consider both libvirt and a user hotplugging something into the same pci slot, or a user starting migration, or quitting, or ...
Having -monitor readonly makes sense.
I don't like such a concept. If a user goes to the trouble to create a second monitor, then they need to know what they're doing.
libvirt can simply not expose the option to the user. Clever users can change the emulator tag to point to a script that they can use to create a second monitor.
Regards,
Anthony Liguori
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
In reply to Jan KiszkaQuoted 33 lines · Jan Kiszka
Quoted 26 lines · Daniel P. Berrange
Quoted 18 lines · Jan Kiszka
Quoted 17 lines · Luiz Capitulino
This change adds a flag called 'control' to the already existing '-monitor' command-line option. This flag can be used to enable control mode.
Its syntax is:
qemu [...] -monitor control,<device>
Where <device> is a chardev (excluding 'vc', for obvious reasons).
For example:
$ qemu [...] -monitor control,tcp:localhost:4444,serverWill run QEMU in control mode, waiting for a client TCP connection on localhost port 4444.
Quoted 6 lines
Quoted 1 line
Signed-off-by Luiz Capitulino <[email protected]>At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor instances". I think Anthony posted such a patch before. Now we should really include this as your extension may block the stand-alone -monitor switch for the control channel, preventing to additionally set up one (or more) for debugging purposes.
Quoted 5 lines
If we support multiple monitors, it might be desirable to allow the app owning the main 'control' monitor channel to be able to indicate that additional monitor channels are read-only. eg, so libvirt could allow the user to connect to the monitor to run 'info' commands for debug support without risk of having state changed behind its back
Quoted 4 lines
Couldn't libvirt deal with update given we provide them as events? Otherwise, your suggestion makes sense, definitely as long as it would wreck libvirt's internal house keeping or for commands that are not synchronizable.
If the mgmt app knows about & supports all features in the QEMU its talking to, and we generate events for all possible changes, then it could detect & cope with changes. In reality though I don't think you can expect mgmt apps to have complete coverage of all features, so it would be safer for them to be able to designate additional monitor channels readonly.
Daniel
Signature · 4 lines
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 37 lines · Daniel P. Berrange
Quoted 31 lines · Jan Kiszka
Quoted 25 lines · Daniel P. Berrange
Quoted 18 lines · Jan Kiszka
Quoted 17 lines · Luiz Capitulino
This change adds a flag called 'control' to the already existing '-monitor' command-line option. This flag can be used to enable control mode.
Its syntax is:
qemu [...] -monitor control,<device>
Where <device> is a chardev (excluding 'vc', for obvious reasons).
For example:
$ qemu [...] -monitor control,tcp:localhost:4444,serverWill run QEMU in control mode, waiting for a client TCP connection on localhost port 4444.
Quoted 6 lines
Quoted 1 line
Signed-off-by Luiz Capitulino <[email protected]>At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor instances". I think Anthony posted such a patch before. Now we should really include this as your extension may block the stand-alone -monitor switch for the control channel, preventing to additionally set up one (or more) for debugging purposes.
Quoted 5 lines
If we support multiple monitors, it might be desirable to allow the app owning the main 'control' monitor channel to be able to indicate that additional monitor channels are read-only. eg, so libvirt could allow the user to connect to the monitor to run 'info' commands for debug support without risk of having state changed behind its back
Quoted 4 lines
Couldn't libvirt deal with update given we provide them as events? Otherwise, your suggestion makes sense, definitely as long as it would wreck libvirt's internal house keeping or for commands that are not synchronizable.
Quoted 6 lines
If the mgmt app knows about & supports all features in the QEMU its talking to, and we generate events for all possible changes, then it could detect & cope with changes. In reality though I don't think you can expect mgmt apps to have complete coverage of all features, so it would be safer for them to be able to designate additional monitor channels readonly.
Also reading your valid concerns regarding event queue overflows, I think making it flexible /wrt asynchronous changes will actually make the whole think much more robust than simply trying to block every competing change.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
In reply to Jan KiszkaQuoted 18 lines · On Tue, 23 Jun 2009 11:03:09 +0200 Jan Kiszka <[email protected]> wrote:
Quoted 17 lines · Luiz Capitulino
This change adds a flag called 'control' to the already existing '-monitor' command-line option. This flag can be used to enable control mode.
Its syntax is:
qemu [...] -monitor control,<device>
Where <device> is a chardev (excluding 'vc', for obvious reasons).
For example:
$ qemu [...] -monitor control,tcp:localhost:4444,serverWill run QEMU in control mode, waiting for a client TCP connection on localhost port 4444.
Quoted 7 lines
Quoted 1 line
Signed-off-by Luiz Capitulino <[email protected]>At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor instances". I think Anthony posted such a patch before. Now we should really include this as your extension may block the stand-alone -monitor switch for the control channel, preventing to additionally set up one (or more) for debugging purposes.
Do you really think that this is needed for the first version? I would
rather prefer to concentrate on protocol specification and command porting, which will require a lot of effort.
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 28 lines · Luiz Capitulino
Quoted 18 lines · On Tue, 23 Jun 2009 11:03:09 +0200 Jan Kiszka <[email protected]> wrote:
Quoted 17 lines · Luiz Capitulino
This change adds a flag called 'control' to the already existing '-monitor' command-line option. This flag can be used to enable control mode.
Its syntax is:
qemu [...] -monitor control,<device>
Where <device> is a chardev (excluding 'vc', for obvious reasons).
For example:
$ qemu [...] -monitor control,tcp:localhost:4444,serverWill run QEMU in control mode, waiting for a client TCP connection on localhost port 4444.
Quoted 6 lines
Quoted 1 line
Signed-off-by Luiz Capitulino <[email protected]>At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor instances". I think Anthony posted such a patch before. Now we should really include this as your extension may block the stand-alone -monitor switch for the control channel, preventing to additionally set up one (or more) for debugging purposes.
Quoted 3 lines
Do you really think that this is needed for the first version? I would
rather prefer to concentrate on protocol specification and command porting, which will require a lot of effort.
You may postpone it if it's too much work, but please have a look at
http://permalink.gmane.org/gmane.comp.emulators.libvirt/12895 first as I think it should be easy to integrate.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
[Qemu-devel] Re: [PATCH 11/11] QMP: Command-line flag to enable control mode
Quoted 29 lines · On Tue, 23 Jun 2009 16:06:14 +0200 Jan Kiszka <[email protected]> wrote:
Quoted 28 lines · Luiz Capitulino
Quoted 24 lines · On Tue, 23 Jun 2009 11:03:09 +0200 Jan Kiszka <[email protected]> wrote:
Quoted 10 lines · Luiz Capitulino
This change adds a flag called 'control' to the already existing '-monitor' command-line option. This flag can be used to enable control mode.
Its syntax is:
qemu [...] -monitor control,<device>
Where <device> is a chardev (excluding 'vc', for obvious reasons).
Quoted 8 lines
For example:
$ qemu [...] -monitor control,tcp:localhost:4444,serverWill run QEMU in control mode, waiting for a client TCP connection on localhost port 4444.
Signed-off-by Luiz Capitulino <[email protected]>At this chance, I would vote for "[PATCH 12/11] Allow multiple -monitor instances". I think Anthony posted such a patch before. Now we should really include this as your extension may block the stand-alone -monitor switch for the control channel, preventing to additionally set up one (or more) for debugging purposes.
Quoted 7 lines
Quoted 3 lines
Do you really think that this is needed for the first version? I would
rather prefer to concentrate on protocol specification and command porting, which will require a lot of effort.You may postpone it if it's too much work, but please have a look at
http://permalink.gmane.org/gmane.comp.emulators.libvirt/12895 first as I think it should be easy to integrate.
I will have a look before deciding, if it doesn't seem that complicated
I will add the patch.
Thanks for the pointer!