qapi: add generator for Golang interface
cover qapi: add generator for Golang interface
Hi,
Happy 1st April. Not a joke :) /* ugh, took me too long to send */
This series is about adding a generator in scripts/qapi to produce Go data structures that can be used to communicate with QEMU over QMP.
* Why Go?
There are quite a few Go projects that interact with QEMU over QMP and they endup using a mix of different libraries with their own code.
** Which projects?
The ones I've found so far:
- podman machine
https://github.com/containers/podman/tree/main/pkg/machine/qemu
- kata-containers (govmm)
https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm
- lxd
https://github.com/lxc/lxd/tree/master/lxd/instance/drivers
- kubevirt (plain json strings)
https://github.com/kubevirt/kubevirt
(let me know if you know others)
* But Why?
I'm particularly interested in 3 out of 4 of the projects above and only Kubevirt uses libvirt to handle QEMU. That means that every QEMU releases where a QMP command, event or other data struct is added, removed or changed, those projects need to check what changed in QEMU and then address those changes in their projects, if needed.
The idea behind generating Go data structures is that we can keep a Go module which can have releases that follow QEMU releases.
The project that uses this Go module, only need to bump the module version and it shall receive all the changes in their own vendored code base.
* Status
There are a few rough edges to work on but this is usable. The major thing I forgot to add is handling Error from Commands. It'll be the first thing I'll work on next week.
If you want to start using this Today you can fetch it in at
https://gitlab.com/victortoso/qapi-go/
There are quite a few tests that I took from the examples in the qapi schema. Coverage using go's cover tool is giving 28.6% of statements
I've uploaded the a static generated godoc output of the above Go module here:
https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/
* License
While the generator (golang.py in this series) is GPL v2, the generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.
* Disclaimer to reviewers
This is my first serious python project so there'll be lots of suggetions that I'll be happy to take and learn from.
Thanks for taking a look, let me know if you have questions, ideas or suggestions.
Cheers,
Victor
Victor Toso (8):
qapi: golang: Generate qapi's enum types in Go
qapi: golang: Generate qapi's alternate types in Go
qapi: golang: Generate qapi's struct types in Go
qapi: golang: Generate qapi's union types in Go
qapi: golang: Generate qapi's event types in Go
qapi: golang: Generate qapi's command types in Go
qapi: golang: Add CommandResult type to Go
qapi: golang: document skip function visit_array_types
qapi/meson.build | 1 + scripts/qapi/golang.py | 727 +++++++++++++++++++++++++++++++++++++++++ scripts/qapi/main.py | 2 + 3 files changed, 730 insertions(+) create mode 100644 scripts/qapi/golang.py
Signature · 1 line
2.35.1
RFC PATCH 1/8 qapi: golang: Generate qapi's enum types in Go
This patch handles QAPI enum types and generates its equivalent in Go.
The highlights of this implementation are:
1. For each QAPI enum, we will define an int32 type in Go to be the assigned type of this specific enum
2. While in the Go codebase we can use the generated enum values, the specification requires that, on the wire, the enumeration type's value to be represented by its string name.
For this reason, each Go type that represent's a QAPI enum will be implementing the Marshaler[0] and Unmarshaler[1] interfaces to seamless handle QMP's string to Go int32 value and vice-versa.
3. Naming: CamelCase will be used in any identifier that we want to export [2], which is everything in this patch.
[0] https://pkg.go.dev/encoding/json#Marshaler
[1] https://pkg.go.dev/encoding/json#Unmarshaler
[2] https://go.dev/ref/spec#Exported_identifiers
--- qapi/meson.build | 1 + scripts/qapi/golang.py | 225 +++++++++++++++++++++++++++++++++++++++++ scripts/qapi/main.py | 2 + 3 files changed, 228 insertions(+) create mode 100644 scripts/qapi/golang.py
qapi/meson.build +1 -0
@@ -90,6 +90,7 @@ qapi_nonmodule_outputs = [ 'qapi-init-commands.h', 'qapi-init-commands.c', 'qapi-events.h', 'qapi-events.c', 'qapi-emit-events.c', 'qapi-emit-events.h', + 'qapibara.go', ] # First build all sources
scripts/qapi/golang.py new file +225 -0
225 lines not on this page · raw message
scripts/qapi/main.py +2 -0
@@ -15,6 +15,7 @@ from .common import must_match from .error import QAPIError from .events import gen_events +from .golang import gen_golang from .introspect import gen_introspect from .schema import QAPISchema from .types import gen_types
@@ -54,6 +55,7 @@ def generate(schema_file: str, gen_events(schema, output_dir, prefix) gen_introspect(schema, output_dir, prefix, unmask) + gen_golang(schema, output_dir, prefix) def main() -> int: """
Signature · 1 line
2.35.1
Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go
Quoted 6 lines · Victor Toso
This patch handles QAPI enum types and generates its equivalent in Go.
The highlights of this implementation are:
1. For each QAPI enum, we will define an int32 type in Go to be the assigned type of this specific enum
I think there's a potential case to be made for considering representation of enums as strings in Golang, as it more gracefully degrades.
Consider the 'SHUTDOWN' event in QEMU, which has a 'reason' field.
This implementation has
type ShutdownCause int32
const (
ShutdownCauseNone ShutdownCause = iota
ShutdownCauseHostError // An error prevents further use of guest
ShutdownCauseHostQmpQuit // Reaction to the QMP command 'quit'
ShutdownCauseHostQmpSystemReset // Reaction to the QMP command 'system_reset'
ShutdownCauseHostSignal // Reaction to a signal, such as SIGINT
ShutdownCauseHostUi // Reaction to a UI event, like window close
ShutdownCauseGuestShutdown // Guest shutdown/suspend request, via ACPI or other hardware-specific means
ShutdownCauseGuestReset // Guest reset request, and command line turns that into a shutdown
ShutdownCauseGuestPanic // Guest panicked, and command line turns that into a shutdown
ShutdownCauseSubsystemReset // Partial guest reset that does not trigger QMP events and ignores --no-reboot. This is useful for sanitizing hypercalls on s390 that are used during kexec/kdump/boot
)
and
type ShutdownEvent struct {
Guest bool json:"guest" // If true, the shutdown was triggered by a guest request (such as a guest-initiated ACPI shutdown request or other hardware-specific action) rather than a host request (such as sending qemu a SIGINT). (since 2.10)
Reason ShutdownCause json:"reason" // The @ShutdownCause which resulted in the SHUTDOWN. (since 4.0)
}
Consider that the application is compiled against the QAPI generated API from QEMU 7.0. The application is believed to run happily against 7.1, because app doesn't need any of the functionality added in 7.1 QAPI. But consider that QEMU 7.1 had introduced a new shutdown reason value.
The application may want to know that the guest has shutdown, but does NOT care about the reason for the shutdown.
Since the ShutdownReason is implemented as an int though, the unmarshalling for the reason field is going to fail.
If the enums were just represented as strings, then we can gracefully accept any new enum value that arrives in future. The application can thus also still log the shutdown reason string, even though it was not a value explicitly known to the generated API.
Quoted 11 lines
2. While in the Go codebase we can use the generated enum values, the specification requires that, on the wire, the enumeration type's value to be represented by its string name.
For this reason, each Go type that represent's a QAPI enum will be implementing the Marshaler[0] and Unmarshaler[1] interfaces to seamless handle QMP's string to Go int32 value and vice-versa.
3. Naming: CamelCase will be used in any identifier that we want to export [2], which is everything in this patch.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go
Hi,
Quoted 44 lines · Daniel P. Berrangé
Quoted 6 lines · Victor Toso
This patch handles QAPI enum types and generates its equivalent in Go.
The highlights of this implementation are:
1. For each QAPI enum, we will define an int32 type in Go to be the assigned type of this specific enum
I think there's a potential case to be made for considering representation of enums as strings in Golang, as it more gracefully degrades.
Consider the 'SHUTDOWN' event in QEMU, which has a 'reason' field.
This implementation has
type ShutdownCause int32
const (
ShutdownCauseNone ShutdownCause = iota
ShutdownCauseHostError // An error prevents further use of guest
ShutdownCauseHostQmpQuit // Reaction to the QMP command 'quit'
ShutdownCauseHostQmpSystemReset // Reaction to the QMP command 'system_reset'
ShutdownCauseHostSignal // Reaction to a signal, such as SIGINT
ShutdownCauseHostUi // Reaction to a UI event, like window close
ShutdownCauseGuestShutdown // Guest shutdown/suspend request, via ACPI or other hardware-specific means
ShutdownCauseGuestReset // Guest reset request, and command line turns that into a shutdown
ShutdownCauseGuestPanic // Guest panicked, and command line turns that into a shutdown
ShutdownCauseSubsystemReset // Partial guest reset that does not trigger QMP events and ignores --no-reboot. This is useful for sanitizing hypercalls on s390 that are used during kexec/kdump/boot
)and
type ShutdownEvent struct {
Guest booljson:"guest"// If true, the shutdown was triggered by a guest request (such as a guest-initiated ACPI shutdown request or other hardware-specific action) rather than a host request (such as sending qemu a SIGINT). (since 2.10)
Reason ShutdownCausejson:"reason"// The @ShutdownCause which resulted in the SHUTDOWN. (since 4.0)
}Consider that the application is compiled against the QAPI generated API from QEMU 7.0. The application is believed to run happily against 7.1, because app doesn't need any of the functionality added in 7.1 QAPI. But consider that QEMU 7.1 had introduced a new shutdown reason value.
Quoted 5 lines
The application may want to know that the guest has shutdown, but does NOT care about the reason for the shutdown.
Since the ShutdownReason is implemented as an int though, the unmarshalling for the reason field is going to fail.
Marshalling does error if you try to convert an int that is not in the range of the enum type.
Unmarshalling should not error in this case, but the field ends up not being set which defaults to 0 (in this case, ShutdownCauseNone). That could mislead the *actual* reason but not without a warning which is expected in this case, imho.
(I know is not an actual warning, just a Println, but it'll be fixed in the next iteration)
| func (s *ShutdownCause) UnmarshalJSON(data []byte) error {
| var name string
|
| if err := json.Unmarshal(data, &name); err != nil {
| return err
| }
|
| switch name {
| case "none":
| (*s) = ShutdownCauseNone
| case "host-error":
| (*s) = ShutdownCauseHostError
| case "host-qmp-quit":
| (*s) = ShutdownCauseHostQmpQuit
| case "host-qmp-system-reset":
| (*s) = ShutdownCauseHostQmpSystemReset
| case "host-signal":
| (*s) = ShutdownCauseHostSignal
| case "host-ui":
| (*s) = ShutdownCauseHostUi
| case "guest-shutdown":
| (*s) = ShutdownCauseGuestShutdown
| case "guest-reset":
| (*s) = ShutdownCauseGuestReset
| case "guest-panic":
| (*s) = ShutdownCauseGuestPanic
| case "subsystem-reset":
| (*s) = ShutdownCauseSubsystemReset
| default:
| fmt.Println("Failed to decode ShutdownCause", *s)
| }
| return nil
| }Quoted 5 lines
If the enums were just represented as strings, then we can gracefully accept any new enum value that arrives in future. The application can thus also still log the shutdown reason string, even though it was not a value explicitly known to the generated API.
As a string, the warning should still exist and default value of "" or nil for ptr would apply. IMHO, between string + warning and int + warning, I'd still go with int here.
That's a design decision to be made. All in all, I don't know much about the changes in QEMU/QAPI per version to take this decision, so I'll rely on you and the list for this, not just for enums but for the other types too.
Quoted 10 lines
Quoted 10 lines
2. While in the Go codebase we can use the generated enum values, the specification requires that, on the wire, the enumeration type's value to be represented by its string name.
For this reason, each Go type that represent's a QAPI enum will be implementing the Marshaler[0] and Unmarshaler[1] interfaces to seamless handle QMP's string to Go int32 value and vice-versa.
3. Naming: CamelCase will be used in any identifier that we want to export [2], which is everything in this patch.
Cheers,
Victor
Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go
Quoted 56 lines · Victor Toso
Hi,
Quoted 44 lines · Daniel P. Berrangé
Quoted 6 lines · Victor Toso
This patch handles QAPI enum types and generates its equivalent in Go.
The highlights of this implementation are:
1. For each QAPI enum, we will define an int32 type in Go to be the assigned type of this specific enum
I think there's a potential case to be made for considering representation of enums as strings in Golang, as it more gracefully degrades.
Consider the 'SHUTDOWN' event in QEMU, which has a 'reason' field.
This implementation has
type ShutdownCause int32
const (
ShutdownCauseNone ShutdownCause = iota
ShutdownCauseHostError // An error prevents further use of guest
ShutdownCauseHostQmpQuit // Reaction to the QMP command 'quit'
ShutdownCauseHostQmpSystemReset // Reaction to the QMP command 'system_reset'
ShutdownCauseHostSignal // Reaction to a signal, such as SIGINT
ShutdownCauseHostUi // Reaction to a UI event, like window close
ShutdownCauseGuestShutdown // Guest shutdown/suspend request, via ACPI or other hardware-specific means
ShutdownCauseGuestReset // Guest reset request, and command line turns that into a shutdown
ShutdownCauseGuestPanic // Guest panicked, and command line turns that into a shutdown
ShutdownCauseSubsystemReset // Partial guest reset that does not trigger QMP events and ignores --no-reboot. This is useful for sanitizing hypercalls on s390 that are used during kexec/kdump/boot
)and
type ShutdownEvent struct {
Guest booljson:"guest"// If true, the shutdown was triggered by a guest request (such as a guest-initiated ACPI shutdown request or other hardware-specific action) rather than a host request (such as sending qemu a SIGINT). (since 2.10)
Reason ShutdownCausejson:"reason"// The @ShutdownCause which resulted in the SHUTDOWN. (since 4.0)
}Consider that the application is compiled against the QAPI generated API from QEMU 7.0. The application is believed to run happily against 7.1, because app doesn't need any of the functionality added in 7.1 QAPI. But consider that QEMU 7.1 had introduced a new shutdown reason value.
Quoted 5 lines
The application may want to know that the guest has shutdown, but does NOT care about the reason for the shutdown.
Since the ShutdownReason is implemented as an int though, the unmarshalling for the reason field is going to fail.
Marshalling does error if you try to convert an int that is not in the range of the enum type.
Quoted 7 lines
Unmarshalling should not error in this case, but the field ends up not being set which defaults to 0 (in this case, ShutdownCauseNone). That could mislead the *actual* reason but not without a warning which is expected in this case, imho.
(I know is not an actual warning, just a Println, but it'll be fixed in the next iteration)
I don't thinnk we should be emitting warnings/printlns at all in this case though. The app should be able to consume the enum without needing a warning. If the app wants to validate against a specific set of constants, it can decide to emit a warning if there was a case it can't handle. We shouldn't emit warnings/errors in the unmarshalling step though,as it isn't the right place to decide on such policy.
Quoted 41 lines
| func (s *ShutdownCause) UnmarshalJSON(data []byte) error { | var name string | | if err := json.Unmarshal(data, &name); err != nil { | return err | } | | switch name { | case "none": | (*s) = ShutdownCauseNone | case "host-error": | (*s) = ShutdownCauseHostError | case "host-qmp-quit": | (*s) = ShutdownCauseHostQmpQuit | case "host-qmp-system-reset": | (*s) = ShutdownCauseHostQmpSystemReset | case "host-signal": | (*s) = ShutdownCauseHostSignal | case "host-ui": | (*s) = ShutdownCauseHostUi | case "guest-shutdown": | (*s) = ShutdownCauseGuestShutdown | case "guest-reset": | (*s) = ShutdownCauseGuestReset | case "guest-panic": | (*s) = ShutdownCauseGuestPanic | case "subsystem-reset": | (*s) = ShutdownCauseSubsystemReset | default: | fmt.Println("Failed to decode ShutdownCause", *s) | } | return nil | }Quoted 5 lines
If the enums were just represented as strings, then we can gracefully accept any new enum value that arrives in future. The application can thus also still log the shutdown reason string, even though it was not a value explicitly known to the generated API.
Quoted 8 lines
As a string, the warning should still exist and default value of "" or nil for ptr would apply. IMHO, between string + warning and int + warning, I'd still go with int here.
That's a design decision to be made. All in all, I don't know much about the changes in QEMU/QAPI per version to take this decision, so I'll rely on you and the list for this, not just for enums but for the other types too.
Essentially every release of QEMU will have QAPI changes. Most of the time these are append-only changes. ie a new struct, new command, new field, new enum value. Sometimes there will be removals due to deprecation, though note my other reply saying that we really ought to stop doing removals from the schema, and instead just annotate when a field stops being used.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go
Hi,
Quoted 11 lines · Daniel P. Berrangé
Quoted 10 lines
Marshalling does error if you try to convert an int that is not in the range of the enum type.
Unmarshalling should not error in this case, but the field ends up not being set which defaults to 0 (in this case, ShutdownCauseNone). That could mislead the *actual* reason but not without a warning which is expected in this case, imho.
(I know is not an actual warning, just a Println, but it'll be fixed in the next iteration)
Quoted 7 lines
I don't thinnk we should be emitting warnings/printlns at all in this case though. The app should be able to consume the enum without needing a warning. If the app wants to validate against a specific set of constants, it can decide to emit a warning if there was a case it can't handle. We shouldn't emit warnings/errors in the unmarshalling step though,as it isn't the right place to decide on such policy.
I think it is useful to know that, a binary compiled to qapi-go 7.0 but running with qemu 7.1 would have some mismatches. It could help detect issues (e.g: Why my program doesn't know/show the reason for shutdown?).
So, some sort of --verbose option for this level should exist.
Quoted 49 lines
Quoted 40 lines
| func (s *ShutdownCause) UnmarshalJSON(data []byte) error { | var name string | | if err := json.Unmarshal(data, &name); err != nil { | return err | } | | switch name { | case "none": | (*s) = ShutdownCauseNone | case "host-error": | (*s) = ShutdownCauseHostError | case "host-qmp-quit": | (*s) = ShutdownCauseHostQmpQuit | case "host-qmp-system-reset": | (*s) = ShutdownCauseHostQmpSystemReset | case "host-signal": | (*s) = ShutdownCauseHostSignal | case "host-ui": | (*s) = ShutdownCauseHostUi | case "guest-shutdown": | (*s) = ShutdownCauseGuestShutdown | case "guest-reset": | (*s) = ShutdownCauseGuestReset | case "guest-panic": | (*s) = ShutdownCauseGuestPanic | case "subsystem-reset": | (*s) = ShutdownCauseSubsystemReset | default: | fmt.Println("Failed to decode ShutdownCause", *s) | } | return nil | }Quoted 5 lines
If the enums were just represented as strings, then we can gracefully accept any new enum value that arrives in future. The application can thus also still log the shutdown reason string, even though it was not a value explicitly known to the generated API.
Quoted 8 lines
As a string, the warning should still exist and default value of "" or nil for ptr would apply. IMHO, between string + warning and int + warning, I'd still go with int here.
That's a design decision to be made. All in all, I don't know much about the changes in QEMU/QAPI per version to take this decision, so I'll rely on you and the list for this, not just for enums but for the other types too.
Quoted 6 lines
Essentially every release of QEMU will have QAPI changes. Most of the time these are append-only changes. ie a new struct, new command, new field, new enum value. Sometimes there will be removals due to deprecation, though note my other reply saying that we really ought to stop doing removals from the schema, and instead just annotate when a field stops being used.
Ok, thanks.
Cheers,
Victor
Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go
Quoted 22 lines · Victor Toso
Hi,
Quoted 11 lines · Daniel P. Berrangé
Quoted 10 lines
Marshalling does error if you try to convert an int that is not in the range of the enum type.
Unmarshalling should not error in this case, but the field ends up not being set which defaults to 0 (in this case, ShutdownCauseNone). That could mislead the *actual* reason but not without a warning which is expected in this case, imho.
(I know is not an actual warning, just a Println, but it'll be fixed in the next iteration)
Quoted 7 lines
I don't thinnk we should be emitting warnings/printlns at all in this case though. The app should be able to consume the enum without needing a warning. If the app wants to validate against a specific set of constants, it can decide to emit a warning if there was a case it can't handle. We shouldn't emit warnings/errors in the unmarshalling step though,as it isn't the right place to decide on such policy.
Quoted 4 lines
I think it is useful to know that, a binary compiled to qapi-go 7.0 but running with qemu 7.1 would have some mismatches. It could help detect issues (e.g: Why my program doesn't know/show the reason for shutdown?).
If apps are actually looking at the shutdown reasons, it is trivial for the app themselves to log any unexpected reasons.
They should in fact do that regardless, because if someone re-compiles the app against a new version of the QEMU Go API, there may be new ShutdownCause's defined that their codes does not handle. There's no compile time check from Go side that their code has handled all ShutdownCause values, since there's no struct enum type in Go. This again pushes me to towards the view that the enums should just remain as strings, the use of integers gives no functional benefit AFAICT, given the lack of a real enum type in Go.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
RFC PATCH 2/8 qapi: golang: Generate qapi's alternate types in Go
In reply to Victor TosoThis patch handles QAPI alternate types and generates data structures in Go that handles it.
At this moment, there are 5 alternates in qemu/qapi, they are:
* BlockDirtyBitmapMergeSource
* Qcow2OverlapChecks
* BlockdevRef
* BlockdevRefOrNull
* StrOrNull
Alternate types are similar to Union but without a discriminator that can be used to identify the underlying value on the wire. It is needed to infer it. That can't be easily mapped in Go.
For each Alternate type, we will be using a Any type to hold the value. 'Any' is an alias type for interface{} (similar to void* in C).
Similarly to the Enum types (see previous commit), we will implement Marshaler and Unmarshaler interfaces for the Alternate types and in those MarshalJSON() and UnmarshalJSON() methods is where we are going to put the logic to read/set alternate's value.
Note that on UnmarshalJSON(), a helper function called StrictDecode() will be used. This function is the main logic to infer if a given JSON object fits in a given Go struct. Because we only have 5 alternate types, it is not hard to validate the unmarshaling logic but we might need to improve it in the future if Alternate with branches that have similar fields appear.
Examples:
* BlockdevRef
```go
// Data to set in BlockdevOptions
qcow2 := BlockdevOptionsQcow2{}
// BlockdevRef using a string
qcow2.File = BlockdevRef{Value: "/some/place/my-image"}
opt := BlockdevOptions{}
opt.Driver = BlockdevDriverQcow2
opt.Value = qcow2
b, _ := json.Marshal(data.s)
// string(b) == `{"driver":"qcow2","file":"/some/place/my-image"}`
```--- scripts/qapi/golang.py | 157 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 155 insertions(+), 2 deletions(-)
scripts/qapi/golang.py +155 -2
@@ -31,7 +31,8 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, prefix: str): super().__init__() - self.target = {name: "" for name in ["enum"]} + self.target = {name: "" for name in ["alternate", "enum", "helper"]} + self.objects_seen = {} self.schema = None self._docmap = {} self.golang_package_name = "qapi"
@@ -43,6 +44,10 @@ def visit_begin(self, schema): for target in self.target: self.target[target] = f"package {self.golang_package_name}\n" + self.target["helper"] += f''' + // Alias for go version lower than 1.18 + type Any = interface{{}}''' + # Iterate once in schema.docs to map doc objects to its name for doc in schema.docs: if doc.symbol is None:
@@ -52,6 +57,22 @@ def visit_begin(self, schema): def visit_end(self): self.schema = None + self.target["helper"] += ''' +// Creates a decoder that errors on unknown Fields +// Returns true if successfully decoded @from string @into type +// Returns false without error is failed with "unknown field" +// Returns false with error is a different error was found +func StrictDecode(into interface{}, from []byte) error { + dec := json.NewDecoder(strings.NewReader(string(from))) + dec.DisallowUnknownFields() + + if err := dec.Decode(into); err != nil { + return err + } + return nil +} +''' + def visit_object_type(self: QAPISchemaGenGolangVisitor, name: str, info: Optional[QAPISourceInfo],
@@ -70,7 +91,123 @@ def visit_alternate_type(self: QAPISchemaGenGolangVisitor, features: List[QAPISchemaFeature], variants: QAPISchemaVariants ) -> None: - pass + assert name not in self.objects_seen + self.objects_seen[name] = True + + # Alternate marshal logic + # + # To avoid programming errors by users of this generated Go module, + # we add a runtime check to error out in case the underlying Go type + # doesn't not match any of supported types of the Alternate type. + # + # Also, Golang's json Marshal will include as JSON's object, the + # wrapper we use to hold the Go struct (Value Any -> `Value: {...}`) + # This would not be an valid QMP message so we workaround it by + # calling RemoveValueObject function. + doc = self._docmap.get(name, None) + doc_struct, doc_fields = qapi_to_golang_struct_docs(doc) + + members_doc = '''// Options are:''' + if_supported_types = "" + for var in variants.variants: + field_doc = doc_fields.get(var.name, "") + field_go_type = qapi_schema_type_to_go_type(var.type.name) + members_doc += f'''\n// * {var.name} ({field_go_type}):{field_doc[3:]}''' + + if field_go_type == "nil": + field_go_type = "*string" + + if_supported_types += f'''typestr != "{field_go_type}" &&\n\t\t''' + + # Alternate unmarshal logic + # + # With Alternate types, we have to check the JSON data in order to + # identify what is the target Go type. So, this is different than an + # union which has an identifier that we can check first. + # StrictDecode function tries to match the given JSON data to a given + # Go type and it'll error in case it doesn´t fit, for instance, when + # there were members in the JSON data that had no equivalent in the + # target Go type. + # + # For this reason, the order is important. + # + # At this moment, the only field that must be checked first is JSON + # NULL, which is relevant to a few alternate types. In the future, we + # might need to improve the logic to be foolproof between target Go + # types that might have a common base (non existing Today). + check_type_str = ''' + // Check for {name} + {{ + var value {go_type} + if err := StrictDecode(&value, data); {error_check} {{ + s.Value = {set_value} + return nil + }} + }}''' + reference_checks = "" + for var in variants.variants: + if var.type.name == "null": + # We use a pointer (by referece) to check for JSON's NULL + reference_checks += check_type_str.format( + name = var.type.name, + go_type = "*string", + error_check = "err == nil && value == nil", + set_value = "nil") + break; + + value_checks = "" + for var in variants.variants: + if var.type.name != "null": + go_type = qapi_schema_type_to_go_type(var.type.name) + value_checks += check_type_str.format( + name = var.type.name, + go_type = go_type, + error_check = "err == nil", + set_value = "value") + + unmarshal_checks = "" + if len(reference_checks) > 0 and len(value_checks) > 0: + unmarshal_checks = reference_checks[1:] + value_checks + else: + unmarshal_checks = reference_checks[1:] if len(reference_checks) > 0 else value_checks[1:] + + self.target["alternate"] += f''' +{doc_struct} +type {name} struct {{ +{members_doc} + Value Any +}} + +func (s {name}) MarshalJSON() ([]byte, error) {{ + typestr := fmt.Sprintf("%T", s.Value) + typestr = typestr[strings.LastIndex(typestr, ".")+1:] + + // Runtime check for supported types + if typestr != "<nil>" && +{if_supported_types[:-6]} {{ + return nil, errors.New(fmt.Sprintf("Type is not supported: %s", typestr)) + }} + + b, err := json.Marshal(s.Value); + if err != nil {{ + return nil, err + }} + + return b, nil +}} + +func (s *{name}) UnmarshalJSON(data []byte) error {{ +{unmarshal_checks} + // Check type to error out nicely + {{ + var value Any + if err := json.Unmarshal(data, &value); err != nil {{ + return err + }} + return errors.New(fmt.Sprintf("Unsupported type %T (value: %v)", value, value)) + }} +}} +''' def visit_enum_type(self: QAPISchemaGenGolangVisitor, name: str,
@@ -208,6 +345,22 @@ def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]): return main, fields +def qapi_schema_type_to_go_type(type: str) -> str: + schema_types_to_go = {'str': 'string', 'null': 'nil', 'bool': 'bool', + 'number': 'float64', 'size': 'uint64', 'int': 'int64', 'int8': 'int8', + 'int16': 'int16', 'int32': 'int32', 'int64': 'int64', 'uint8': 'uint8', + 'uint16': 'uint16', 'uint32': 'uint32', 'uint64': 'uint64', + 'any': 'Any', 'QType': 'QType', + } + + prefix = "" + if type.endswith("List"): + prefix = "[]" + type = type[:-4] + + type = schema_types_to_go.get(type, type) + return prefix + type + def qapi_to_field_name_enum(name: str) -> str: return name.title().replace("-", "")
Signature · 1 line
2.35.1
Re: [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate types in Go
Quoted 10 lines · Victor Toso
This patch handles QAPI alternate types and generates data structures in Go that handles it.
At this moment, there are 5 alternates in qemu/qapi, they are:
* BlockDirtyBitmapMergeSource
* Qcow2OverlapChecks
* BlockdevRef
* BlockdevRefOrNull
* StrOrNull
Quoted 3 lines
Alternate types are similar to Union but without a discriminator that can be used to identify the underlying value on the wire. It is needed to infer it. That can't be easily mapped in Go.
I don't buy that. Given this example:
type BlockdevRef struct {
// Options are:
// * definition (BlockdevOptions): defines a new block device inline
// * reference (string): references the ID of an existing block device
Value Any}
What is the problem with having this Go struct:
type BlockdevRef struct {
Definition *BlockdevOptions
Reference *string
}
when deserializing from JSON, we know exactly which one of these two fields to populate. The programmer consuming this can look at which field is non-nil.
When serializing to JSON, we serialize which ever field is non-nil.
If both fields are non-nil that's a programmer bug. Either ignore it and only serialize the first non-nil field, or raise an error.
Quoted 31 lines
For each Alternate type, we will be using a Any type to hold the value. 'Any' is an alias type for interface{} (similar to void* in C).
Similarly to the Enum types (see previous commit), we will implement Marshaler and Unmarshaler interfaces for the Alternate types and in those
MarshalJSON()andUnmarshalJSON()methods is where we are going to put the logic to read/set alternate's value.Note that on
UnmarshalJSON(), a helper function calledStrictDecode()will be used. This function is the main logic to infer if a given JSON object fits in a given Go struct. Because we only have 5 alternate types, it is not hard to validate the unmarshaling logic but we might need to improve it in the future if Alternate with branches that have similar fields appear.Examples:
* BlockdevRef```go // Data to set in BlockdevOptions qcow2 := BlockdevOptionsQcow2{} // BlockdevRef using a string qcow2.File = BlockdevRef{Value: "/some/place/my-image"} opt := BlockdevOptions{} opt.Driver = BlockdevDriverQcow2 opt.Value = qcow2 b, _ := json.Marshal(data.s) // string(b) == `{"driver":"qcow2","file":"/some/place/my-image"}` ```
Quoted 4 lines
Signed-off-by Victor Toso <[email protected]>--- scripts/qapi/golang.py | 157 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 155 insertions(+), 2 deletions(-)
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate types in Go
Hi,
Quoted 23 lines · Daniel P. Berrangé
Quoted 10 lines · Victor Toso
This patch handles QAPI alternate types and generates data structures in Go that handles it.
At this moment, there are 5 alternates in qemu/qapi, they are:
* BlockDirtyBitmapMergeSource
* Qcow2OverlapChecks
* BlockdevRef
* BlockdevRefOrNull
* StrOrNullQuoted 4 lines
Alternate types are similar to Union but without a discriminator that can be used to identify the underlying value on the wire. It is needed to infer it. That can't be easily mapped in Go.
I don't buy that. Given this example:
type BlockdevRef struct { // Options are: // * definition (BlockdevOptions): defines a new block device inline // * reference (string): references the ID of an existing block device Value Any
Quoted 8 lines
}
What is the problem with having this Go struct:
type BlockdevRef struct {
Definition *BlockdevOptions
Reference *string
}
... this is better.
Quoted 10 lines
when deserializing from JSON, we know exactly which one of these two fields to populate. The programmer consuming this can look at which field is non-nil.
When serializing to JSON, we serialize which ever field is non-nil.
If both fields are non-nil that's a programmer bug. Either ignore it and only serialize the first non-nil field, or raise an error.
It would be a programmer bug if they set a Value of a type not allowed by Type's spec, but it would be a *runtime* error. Your suggestion is more type safe.
Thanks.
Quoted 44 lines
Quoted 32 lines
For each Alternate type, we will be using a Any type to hold the value. 'Any' is an alias type for interface{} (similar to void* in C).
Similarly to the Enum types (see previous commit), we will implement Marshaler and Unmarshaler interfaces for the Alternate types and in those
MarshalJSON()andUnmarshalJSON()methods is where we are going to put the logic to read/set alternate's value.Note that on
UnmarshalJSON(), a helper function calledStrictDecode()will be used. This function is the main logic to infer if a given JSON object fits in a given Go struct. Because we only have 5 alternate types, it is not hard to validate the unmarshaling logic but we might need to improve it in the future if Alternate with branches that have similar fields appear.Examples:
* BlockdevRef```go // Data to set in BlockdevOptions qcow2 := BlockdevOptionsQcow2{} // BlockdevRef using a string qcow2.File = BlockdevRef{Value: "/some/place/my-image"} opt := BlockdevOptions{} opt.Driver = BlockdevDriverQcow2 opt.Value = qcow2 b, _ := json.Marshal(data.s) // string(b) == `{"driver":"qcow2","file":"/some/place/my-image"}` ```Quoted 4 lines
Signed-off-by Victor Toso <[email protected]>--- scripts/qapi/golang.py | 157 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 155 insertions(+), 2 deletions(-)With regards,
DanielSignature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate types in Go
Quoted 44 lines · Victor Toso
Hi,
Quoted 23 lines · Daniel P. Berrangé
Quoted 10 lines · Victor Toso
This patch handles QAPI alternate types and generates data structures in Go that handles it.
At this moment, there are 5 alternates in qemu/qapi, they are:
* BlockDirtyBitmapMergeSource
* Qcow2OverlapChecks
* BlockdevRef
* BlockdevRefOrNull
* StrOrNullQuoted 4 lines
Alternate types are similar to Union but without a discriminator that can be used to identify the underlying value on the wire. It is needed to infer it. That can't be easily mapped in Go.
I don't buy that. Given this example:
type BlockdevRef struct { // Options are: // * definition (BlockdevOptions): defines a new block device inline // * reference (string): references the ID of an existing block device Value AnyQuoted 8 lines
}
What is the problem with having this Go struct:
type BlockdevRef struct {
Definition *BlockdevOptions
Reference *string
}... this is better.
Quoted 7 lines
when deserializing from JSON, we know exactly which one of these two fields to populate. The programmer consuming this can look at which field is non-nil.
When serializing to JSON, we serialize which ever field is non-nil.
Quoted 7 lines
Quoted 3 lines
If both fields are non-nil that's a programmer bug. Either ignore it and only serialize the first non-nil field, or raise an error.
It would be a programmer bug if they set a Value of a type not allowed by Type's spec, but it would be a *runtime* error. Your suggestion is more type safe.
Yep, essentially I'm saying I want the code to enable type checking to be done at compile time instead of runtime, whenever it is possible to achieve that from a technical POV.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
RFC PATCH 3/8 qapi: golang: Generate qapi's struct types in Go
In reply to Victor TosoThis patch handles QAPI struct types and generates the equivalent types in Go.
At the time of this writing, it generates 375 structures.
The highlights of this implementation are:
1. Generating an Go struct that requires a @base type, the @base type is embedded in this Go structure.
Example: See InetSocketAddress with it's base InetSocketAddressBase
2. Differently from previous two types ('enum' and 'alternate'), the generated QAPI's struct type do not need to implement Marshaler and Unmarshaler interfaces. This generated structures will naturally match with JSON Objects.
3. About the Go struct's fields:
i) They can be either by Value or Reference.
ii) Every field that is marked as optional in the QAPI specification are translated to Reference fields in its Go structure. This design decision is the most straightforward way to check if a given field was set or not.
iii) Mandatory fields are always by Value with the exception of QAPI arrays, which are handled by Reference (to a block of memory) by Go.
iv) All the fields are named with Uppercase due Golang's export convention.
v) In order to avoid any kind of issues when encoding ordecoding, to or from JSON, we mark all fields with its @name and, when it is optional, member, with @omitempty
--- scripts/qapi/golang.py | 79 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 77 insertions(+), 2 deletions(-)
scripts/qapi/golang.py +77 -2
@@ -31,7 +31,7 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, prefix: str): super().__init__() - self.target = {name: "" for name in ["alternate", "enum", "helper"]} + self.target = {name: "" for name in ["alternate", "enum", "helper", "struct"]} self.objects_seen = {} self.schema = None self._docmap = {}
@@ -82,7 +82,31 @@ def visit_object_type(self: QAPISchemaGenGolangVisitor, members: List[QAPISchemaObjectTypeMember], variants: Optional[QAPISchemaVariants] ) -> None: - pass + # Do not handle anything besides structs + if (name == self.schema.the_empty_object_type.name or + not isinstance(name, str) or + info.defn_meta not in ["struct"]): + return + + assert name not in self.objects_seen + self.objects_seen[name] = True + + # visit all inner objects as well, they are not going to be + # called by python's generator. + if variants: + for var in variants.variants: + assert isinstance(var.type, QAPISchemaObjectType) + self.visit_object_type(self, + var.type.name, + var.type.info, + var.type.ifcond, + var.type.base, + var.type.local_members, + var.type.variants) + + doc = self._docmap.get(info.defn_name, None) + self.target[info.defn_meta] += qapi_to_golang_struct(name, doc, info, + ifcond, features, base, members, variants) def visit_alternate_type(self: QAPISchemaGenGolangVisitor, name: str,
@@ -276,6 +300,14 @@ def gen_golang(schema: QAPISchema, schema.visit(vis) vis.write(output_dir) +# Helper function for boxed or self contained structures. +def generate_struct_type(type_name, args="", doc_struct="") -> str: + args = args if len(args) == 0 else f"\n{args}\n" + return f''' +{doc_struct} +type {type_name} struct {{{args}}} +''' + def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str: type = qapi_to_go_type_name(members[0].defined_in, "enum")
@@ -345,6 +377,46 @@ def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]): return main, fields +# Helper function that is used for most of QAPI types +def qapi_to_golang_struct(name: str, + doc: QAPIDoc, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + base: Optional[QAPISchemaObjectType], + members: List[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants]) -> str: + + type_name = qapi_to_go_type_name(name, info.defn_meta) + doc_struct, doc_fields = qapi_to_golang_struct_docs(doc) + + base_fields = "" + if base: + base_type_name = qapi_to_go_type_name(base.name, base.meta) + base_fields = f"\t// Base type for this struct\n\t{base_type_name}\n" + + own_fields = "" + for memb in members: + field = qapi_to_field_name(memb.name) + member_type = qapi_schema_type_to_go_type(memb.type.name) + + # In Golang, we are using "encoding/json" library to Marshal and Unmarshal between + # over-the-wire QMP and Golang struct. The field tag 'omitempty' does not behave as + # expected for some types with default values and they only way to "ignore by default" + # unset fields is to have them as reference in the Struct. + # This way, a *bool and *int can be ignored where a bool or int might have been set. + isptr = "*" if memb.optional and member_type[0] not in "*[" else "" + optional = ",omitempty" if memb.optional else "" + fieldtag = '`json:"{name}{optional}"`'.format(name=memb.name,optional=optional) + + field_doc = doc_fields.get(memb.name, "") + own_fields += f"\t{field} {isptr}{member_type}{fieldtag}{field_doc}\n" + + all_fields = base_fields if len(base_fields) > 0 else "" + all_fields += own_fields[:-1] if len(own_fields) > 0 else "" + + return generate_struct_type(type_name, all_fields, doc_struct) + def qapi_schema_type_to_go_type(type: str) -> str: schema_types_to_go = {'str': 'string', 'null': 'nil', 'bool': 'bool', 'number': 'float64', 'size': 'uint64', 'int': 'int64', 'int8': 'int8',
@@ -364,6 +436,9 @@ def qapi_schema_type_to_go_type(type: str) -> str: def qapi_to_field_name_enum(name: str) -> str: return name.title().replace("-", "") +def qapi_to_field_name(name: str) -> str: + return name.title().replace("_", "").replace("-", "") + def qapi_to_go_type_name(name: str, meta: str) -> str: if name.startswith("q_obj_"): name = name[6:]
Signature · 1 line
2.35.1
RFC PATCH 4/8 qapi: golang: Generate qapi's union types in Go
In reply to Victor TosoThis patch handles QAPI union types and generates the equivalent data structures and methods in Go to handle it.
At the moment of this writing, it generates 67 structures.
The QAPI union type can be summarized by its common members that are defined in a @base struct and a @value. The @value type can vary and depends on @base's field that we call @discriminator. The @discriminator is always a Enum type.
Golang does not have Unions. The generation of QAPI union type in Go with this patch, follows similar approach to what is done for QAPI struct types and QAPI alternate types.
Similarly to Go implementation of QAPI alternate types, we will implement the Marshaler and Unmarshaler interfaces to seamless decode from JSON objects to Golang structs and vice versa.
Similarly to Go implementation of QAPI struct types, we will need to tag @base fields accordingly.
The embedded documentation in Golang's structures and fields are particularly important here, to help developers know what Types to use for @value. Runtime checks too.
--- scripts/qapi/golang.py | 124 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 119 insertions(+), 5 deletions(-)
scripts/qapi/golang.py +119 -5
@@ -31,7 +31,7 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, prefix: str): super().__init__() - self.target = {name: "" for name in ["alternate", "enum", "helper", "struct"]} + self.target = {name: "" for name in ["alternate", "enum", "helper", "struct", "union"]} self.objects_seen = {} self.schema = None self._docmap = {}
@@ -82,10 +82,10 @@ def visit_object_type(self: QAPISchemaGenGolangVisitor, members: List[QAPISchemaObjectTypeMember], variants: Optional[QAPISchemaVariants] ) -> None: - # Do not handle anything besides structs + # Do not handle anything besides struct and unions. if (name == self.schema.the_empty_object_type.name or not isinstance(name, str) or - info.defn_meta not in ["struct"]): + info.defn_meta not in ["struct", "union"]): return assert name not in self.objects_seen
@@ -351,6 +351,93 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str: }} ''' +# Marshal methods for Union types +def generate_marshal_methods(type: str, + type_dict: Dict[str, str], + discriminator: str = "", + base: str = "") -> str: + assert base != "" + discriminator = "base." + discriminator + + switch_case_format = ''' + case {name}: + value := {case_type}{{}} + if err := json.Unmarshal(data, &value); err != nil {{ + return err + }} + s.Value = value''' + + if_supported_types = "" + added = {} + switch_cases = "" + for name in sorted(type_dict): + case_type = type_dict[name] + isptr = "*" if case_type[0] not in "*[" else "" + switch_cases += switch_case_format.format(name = name, + case_type = case_type) + if case_type not in added: + if_supported_types += f'''typestr != "{case_type}" &&\n\t\t''' + added[case_type] = True + + marshalfn = f''' +func (s {type}) MarshalJSON() ([]byte, error) {{ + base, err := json.Marshal(s.{base}) + if err != nil {{ + return nil, err + }} + + typestr := fmt.Sprintf("%T", s.Value) + typestr = typestr[strings.LastIndex(typestr, ".")+1:] + + // "The branches need not cover all possible enum values" + // This means that on Marshal, we can safely ignore empty values + if typestr == "<nil>" {{ + return []byte(base), nil + }} + + // Runtime check for supported value types + if {if_supported_types[:-6]} {{ + return nil, errors.New(fmt.Sprintf("Type is not supported: %s", typestr)) + }} + value, err := json.Marshal(s.Value) + if err != nil {{ + return nil, err + }} + + // Workaround to avoid checking s.Value being empty + if string(value) == "{{}}" {{ + return []byte(base), nil + }} + + // Removes the last '}}' from base and the first '{{' from value, in order to + // return a single JSON object. + result := fmt.Sprintf("%s,%s", base[:len(base)-1], value[1:]) + return []byte(result), nil +}} +''' + unmarshal_base = f''' + var base {base} + if err := json.Unmarshal(data, &base); err != nil {{ + return err + }} + s.{base} = base +''' + unmarshal_default_warn = f''' + default: + fmt.Println("Failed to decode {type}", {discriminator})''' + + return f'''{marshalfn} +func (s *{type}) UnmarshalJSON(data []byte) error {{ + {unmarshal_base} + switch {discriminator} {{ +{switch_cases[1:]} + {unmarshal_default_warn} + }} + + return nil +}} +''' + # Takes the documentation object of a specific type and returns # that type's documentation followed by its member's docs. def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]):
@@ -412,10 +499,37 @@ def qapi_to_golang_struct(name: str, field_doc = doc_fields.get(memb.name, "") own_fields += f"\t{field} {isptr}{member_type}{fieldtag}{field_doc}\n" + union_types = {} + variant_fields = "" + if variants: + variant_fields = f"// Value based on @{variants.tag_member.name}, possible types:" + for var in variants.variants: + if var.type.is_implicit(): + continue + + name = variants.tag_member._type_name + var.name.title().replace("-", "") + union_types[name] = var.type.name + variant_fields += f"\n\t// * {var.type.c_unboxed_type()}" + + variant_fields += f"\n\tValue Any" + all_fields = base_fields if len(base_fields) > 0 else "" all_fields += own_fields[:-1] if len(own_fields) > 0 else "" - - return generate_struct_type(type_name, all_fields, doc_struct) + all_fields += variant_fields if len(variant_fields) > 0 else "" + + unmarshal_fn = "" + if info.defn_meta == "union" and variants is not None: + # Union's without variants are the Union's base data structure. + # e.g: SchemaInfo's base is SchemainfoBase. + discriminator = qapi_to_field_name(variants.tag_member.name) + base = qapi_to_go_type_name(variants.tag_member.defined_in, + variants.tag_member.info.defn_meta) + unmarshal_fn = generate_marshal_methods(type_name, + union_types, + discriminator = discriminator, + base = base_type_name) + + return generate_struct_type(type_name, all_fields, doc_struct) + unmarshal_fn def qapi_schema_type_to_go_type(type: str) -> str: schema_types_to_go = {'str': 'string', 'null': 'nil', 'bool': 'bool',
Signature · 1 line
2.35.1
Re: [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union types in Go
Quoted 5 lines · Victor Toso
This patch handles QAPI union types and generates the equivalent data structures and methods in Go to handle it.
At the moment of this writing, it generates 67 structures.
Quoted 8 lines
The QAPI union type can be summarized by its common members that are defined in a @base struct and a @value. The @value type can vary and depends on @base's field that we call @discriminator. The @discriminator is always a Enum type.
Golang does not have Unions. The generation of QAPI union type in Go with this patch, follows similar approach to what is done for QAPI struct types and QAPI alternate types.
The common way to approach unions in Go is to just use a struct where each union case is an optional field, and declare that only one field must ever be set. ie
type SocketAddressLegacy struct {
// Value based on @type, possible types:
Inet *InetSocketAddressWrapper
Unix *UnixSocketAddressWrapper
VSock *VsockSocketAddressWrapper
FD *StringWrapper
}When deserializing from JSON we populate exactly one of the optional fields.
When serializing to JSON process the first field that is non-nil.
Note, you don't actually need to include the discriminator as a field at all, since it is implicitly determined by whichever case is non-nil. Introducing the discriminator as a field just provides the possibility for the programmer to make inconsistent settings, for no gain.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union types in Go
Hi,
Quoted 19 lines · Daniel P. Berrangé
Quoted 5 lines · Victor Toso
This patch handles QAPI union types and generates the equivalent data structures and methods in Go to handle it.
At the moment of this writing, it generates 67 structures.
Quoted 8 lines
The QAPI union type can be summarized by its common members that are defined in a @base struct and a @value. The @value type can vary and depends on @base's field that we call @discriminator. The @discriminator is always a Enum type.
Golang does not have Unions. The generation of QAPI union type in Go with this patch, follows similar approach to what is done for QAPI struct types and QAPI alternate types.
The common way to approach unions in Go is to just use a struct where each union case is an optional field, and declare that only one field must ever be set. ie
Quoted 7 lines
type SocketAddressLegacy struct {
// Value based on @type, possible types:
Inet *InetSocketAddressWrapper
Unix *UnixSocketAddressWrapper
VSock *VsockSocketAddressWrapper
FD *StringWrapper
}Like Alternates, I like this better.
Quoted 11 lines
When deserializing from JSON we populate exactly one of the optional fields.
When serializing to JSON process the first field that is non-nil.
Note, you don't actually need to include the discriminator as a field at all, since it is implicitly determined by whichever case is non-nil. Introducing the discriminator as a field just provides the possibility for the programmer to make inconsistent settings, for no gain.
Sounds reasonable. We still need to implement Marshal/Unmarshal for unknow types (e.g: a new Type for SocketAddressLegacy was introduced in 7.1 and we should be able to know that current qapi-go version can't understand it).
Cheers,
Victor
Re: [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union types in Go
Quoted 44 lines · Victor Toso
Hi,
Quoted 19 lines · Daniel P. Berrangé
Quoted 5 lines · Victor Toso
This patch handles QAPI union types and generates the equivalent data structures and methods in Go to handle it.
At the moment of this writing, it generates 67 structures.
Quoted 8 lines
The QAPI union type can be summarized by its common members that are defined in a @base struct and a @value. The @value type can vary and depends on @base's field that we call @discriminator. The @discriminator is always a Enum type.
Golang does not have Unions. The generation of QAPI union type in Go with this patch, follows similar approach to what is done for QAPI struct types and QAPI alternate types.
The common way to approach unions in Go is to just use a struct where each union case is an optional field, and declare that only one field must ever be set. ie
Quoted 7 lines
type SocketAddressLegacy struct { // Value based on @type, possible types: Inet *InetSocketAddressWrapper Unix *UnixSocketAddressWrapper VSock *VsockSocketAddressWrapper FD *StringWrapper }Like Alternates, I like this better.
Quoted 11 lines
When deserializing from JSON we populate exactly one of the optional fields.
When serializing to JSON process the first field that is non-nil.
Note, you don't actually need to include the discriminator as a field at all, since it is implicitly determined by whichever case is non-nil. Introducing the discriminator as a field just provides the possibility for the programmer to make inconsistent settings, for no gain.
Quoted 4 lines
Sounds reasonable. We still need to implement Marshal/Unmarshal for unknow types (e.g: a new Type for SocketAddressLegacy was introduced in 7.1 and we should be able to know that current qapi-go version can't understand it).
If there's a new type seen on the wire then, the easy option is to just not deserialize it at all. Return a SocketAddressLegacy struct with all fields nil, or perhaps return a full 'error', since this is likely to be significant functionalproblem for the application.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
RFC PATCH 5/8 qapi: golang: Generate qapi's event types in Go
In reply to Victor TosoThis patch handles QAPI event types and generates data structures in Go that handles it.
At the moment of this writing, it generates 51 structures (49 events)
In Golang, each event is handled as a Go structure and there is no big difference, in the Go generated code, between what is a QAPI event type and what is a QAPI struct.
Each QAPI event has the suffix 'Event' in its Golang data structure and contains the fields, mandatory and optional, that can be sent or received.
In addition, there are two structures added to handle QAPI specification for event types: 'Event' and 'EventBase'.
'EventBase' contains @Name and @Timestamp fields and then 'Event' extends 'EventBase' with an @Arg field of type 'Any'.
The 'Event' type implements the Unmarshaler to decode the QMP JSON Object into the correct Golang (event) struct. The goal here is to facilitate receiving Events.
A TODO for this type is to implement Marshaler for 'Event'. It'll containt runtime checks to validate before transforming the struct into a JSON Object.
Example:
```go
qmpMsg := `{
"event" : "MIGRATION",
"timestamp":{
"seconds": 1432121972,
"microseconds": 744001
},
"data":{
"status": "completed"
}
}`
e := Event{}
_ = json.Unmarshal([]byte(qmpMsg), &e)
// e.Name == "MIGRATION"
// e.Arg.(MigrationEvent).Status == MigrationStatusCompleted
```--- scripts/qapi/golang.py | 92 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 84 insertions(+), 8 deletions(-)
scripts/qapi/golang.py +84 -8
@@ -31,9 +31,10 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, prefix: str): super().__init__() - self.target = {name: "" for name in ["alternate", "enum", "helper", "struct", "union"]} + self.target = {name: "" for name in ["alternate", "enum", "event", "helper", "struct", "union"]} self.objects_seen = {} self.schema = None + self.events = {} self._docmap = {} self.golang_package_name = "qapi"
@@ -57,6 +58,24 @@ def visit_begin(self, schema): def visit_end(self): self.schema = None + # EventBase and Event are not specified in the QAPI schema, + # so we need to generate it ourselves. + self.target["event"] += ''' +type EventBase struct { + Name string `json:"event"` + Timestamp struct { + Seconds int64 `json:"seconds"` + Microseconds int64 `json:"microseconds"` + } `json:"timestamp"` +} + +type Event struct { + EventBase + Arg Any `json:"data,omitempty"` +} +''' + self.target["event"] += generate_marshal_methods('Event', self.events) + self.target["helper"] += ''' // Creates a decoder that errors on unknown Fields // Returns true if successfully decoded @from string @into type
@@ -279,7 +298,28 @@ def visit_command(self, pass def visit_event(self, name, info, ifcond, features, arg_type, boxed): - pass + assert name == info.defn_name + type_name = qapi_to_go_type_name(name, info.defn_meta) + self.events[name] = type_name + + doc = self._docmap.get(name, None) + self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False + content = "" + if self_contained: + doc_struct, _ = qapi_to_golang_struct_docs(doc) + content = generate_struct_type(type_name, "", doc_struct) + else: + assert isinstance(arg_type, QAPISchemaObjectType) + content = qapi_to_golang_struct(name, + doc, + arg_type.info, + arg_type.ifcond, + arg_type.features, + arg_type.base, + arg_type.members, + arg_type.variants) + + self.target["event"] += content def write(self, output_dir: str) -> None: for module_name, content in self.target.items():
@@ -351,15 +391,41 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str: }} ''' -# Marshal methods for Union types +# Marshal methods for Event and Union types def generate_marshal_methods(type: str, type_dict: Dict[str, str], discriminator: str = "", base: str = "") -> str: - assert base != "" - discriminator = "base." + discriminator - - switch_case_format = ''' + type_is_union = False + json_field = "" + struct_field = "" + if type == "Event": + base = type + "Base" + discriminator = "base.Name" + struct_field = "Arg" + json_field = "data" + else: + assert base != "" + discriminator = "base." + discriminator + type_is_union = True + + switch_case_format = "" + if not type_is_union: + switch_case_format = ''' + case "{name}": + tmp := struct {{ + Value {isptr}{case_type} `json:"{json_field},omitempty"` + }}{{}} + if err := json.Unmarshal(data, &tmp); err != nil {{ + return err + }} + if tmp.Value == nil {{ + s.{struct_field} = nil + }} else {{ + s.{struct_field} = {isptr}tmp.Value + }}''' + else: + switch_case_format = ''' case {name}: value := {case_type}{{}} if err := json.Unmarshal(data, &value); err != nil {{
@@ -374,12 +440,17 @@ def generate_marshal_methods(type: str, case_type = type_dict[name] isptr = "*" if case_type[0] not in "*[" else "" switch_cases += switch_case_format.format(name = name, + struct_field = struct_field, + json_field = json_field, + isptr = isptr, case_type = case_type) if case_type not in added: if_supported_types += f'''typestr != "{case_type}" &&\n\t\t''' added[case_type] = True - marshalfn = f''' + marshalfn = "" + if type_is_union: + marshalfn = f''' func (s {type}) MarshalJSON() ([]byte, error) {{ base, err := json.Marshal(s.{base}) if err != nil {{
@@ -564,4 +635,9 @@ def qapi_to_go_type_name(name: str, meta: str) -> str: words = [word for word in name.replace("_", "-").split("-")] name = words[0].title() if words[0].islower() or words[0].isupper() else words[0] name += ''.join(word.title() for word in words[1:]) + + if meta == "event": + name = name[:-3] if name.endswith("Arg") else name + name += meta.title() + return name
Signature · 1 line
2.35.1
Re: [RFC PATCH v1 5/8] qapi: golang: Generate qapi's event types in Go
Quoted 13 lines · Victor Toso
This patch handles QAPI event types and generates data structures in Go that handles it.
At the moment of this writing, it generates 51 structures (49 events)
In Golang, each event is handled as a Go structure and there is no big difference, in the Go generated code, between what is a QAPI event type and what is a QAPI struct.
Each QAPI event has the suffix 'Event' in its Golang data structure and contains the fields, mandatory and optional, that can be sent or received.
Quoted 5 lines
In addition, there are two structures added to handle QAPI specification for event types: 'Event' and 'EventBase'.
'EventBase' contains @Name and @Timestamp fields and then 'Event' extends 'EventBase' with an @Arg field of type 'Any'.
Again, I don't think we should need to use an Any type here.
Rather than
type EventBase struct {
Name string `json:"event"`
Timestamp struct {
Seconds int64 `json:"seconds"`
Microseconds int64 `json:"microseconds"`
} `json:"timestamp"`
}
type Event struct {
EventBase
Arg Any `json:"data,omitempty"`
}
type ShutdownEvent struct {
Guest bool `json:"guest"`
Reason ShutdownCause `json:"reason"`
}I think we should just embed EventBase directly in each specific event eg
type Event struct {
Name string `json:"event"`
Timestamp struct {
Seconds int64 `json:"seconds"`
Microseconds int64 `json:"microseconds"`
} `json:"timestamp"`
}
type ShutdownEvent struct {
Event Event
Guest bool `json:"guest"`
Reason ShutdownCause `json:"reason"`
}Or perhaps better still, use an interface
type Event interface {
GetName() string
GetTimestamp() string
}
type Timestamp struct {
Seconds int64 `json:"seconds"`
Microseconds int64 `json:"microseconds"`
}
type ShutdownEvent struct {
Timestamp Timestamp `json:"timestamp"`
Guest bool `json:"guest"`
Reason ShutdownCause `json:"reason"`
}
func (ev *ShutdownEvent) GetName() string {
return "SHUTDOWN"
}That way you can define public APIs taking 'Event' as a type, and impls of the events can be passed directly in/out.
Similar comment for the Command type.
Quoted 184 lines
The 'Event' type implements the Unmarshaler to decode the QMP JSON Object into the correct Golang (event) struct. The goal here is to facilitate receiving Events.
A TODO for this type is to implement Marshaler for 'Event'. It'll containt runtime checks to validate before transforming the struct into a JSON Object.
Example:
```go qmpMsg := `{ "event" : "MIGRATION", "timestamp":{ "seconds": 1432121972, "microseconds": 744001 }, "data":{ "status": "completed" } }` e := Event{} _ = json.Unmarshal([]byte(qmpMsg), &e) // e.Name == "MIGRATION" // e.Arg.(MigrationEvent).Status == MigrationStatusCompleted ```Signed-off-by Victor Toso <[email protected]>--- scripts/qapi/golang.py | 92 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 84 insertions(+), 8 deletions(-)scripts/qapi/golang.py +84
-8@@ -31,9 +31,10 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, prefix: str): super().__init__() - self.target = {name: "" for name in ["alternate", "enum", "helper", "struct", "union"]} + self.target = {name: "" for name in ["alternate", "enum", "event", "helper", "struct", "union"]} self.objects_seen = {} self.schema = None + self.events = {} self._docmap = {} self.golang_package_name = "qapi"@@ -57,6 +58,24 @@ def visit_begin(self, schema): def visit_end(self): self.schema = None + # EventBase and Event are not specified in the QAPI schema, + # so we need to generate it ourselves. + self.target["event"] += ''' +type EventBase struct { + Name string `json:"event"` + Timestamp struct { + Seconds int64 `json:"seconds"` + Microseconds int64 `json:"microseconds"` + } `json:"timestamp"` +} + +type Event struct { + EventBase + Arg Any `json:"data,omitempty"` +} +''' + self.target["event"] += generate_marshal_methods('Event', self.events) + self.target["helper"] += ''' // Creates a decoder that errors on unknown Fields // Returns true if successfully decoded @from string @into type@@ -279,7 +298,28 @@ def visit_command(self, pass def visit_event(self, name, info, ifcond, features, arg_type, boxed): - pass + assert name == info.defn_name + type_name = qapi_to_go_type_name(name, info.defn_meta) + self.events[name] = type_name + + doc = self._docmap.get(name, None) + self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False + content = "" + if self_contained: + doc_struct, _ = qapi_to_golang_struct_docs(doc) + content = generate_struct_type(type_name, "", doc_struct) + else: + assert isinstance(arg_type, QAPISchemaObjectType) + content = qapi_to_golang_struct(name, + doc, + arg_type.info, + arg_type.ifcond, + arg_type.features, + arg_type.base, + arg_type.members, + arg_type.variants) + + self.target["event"] += content def write(self, output_dir: str) -> None: for module_name, content in self.target.items():@@ -351,15 +391,41 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str: }} ''' -# Marshal methods for Union types +# Marshal methods for Event and Union types def generate_marshal_methods(type: str, type_dict: Dict[str, str], discriminator: str = "", base: str = "") -> str: - assert base != "" - discriminator = "base." + discriminator - - switch_case_format = ''' + type_is_union = False + json_field = "" + struct_field = "" + if type == "Event": + base = type + "Base" + discriminator = "base.Name" + struct_field = "Arg" + json_field = "data" + else: + assert base != "" + discriminator = "base." + discriminator + type_is_union = True + + switch_case_format = "" + if not type_is_union: + switch_case_format = ''' + case "{name}": + tmp := struct {{ + Value {isptr}{case_type} `json:"{json_field},omitempty"` + }}{{}} + if err := json.Unmarshal(data, &tmp); err != nil {{ + return err + }} + if tmp.Value == nil {{ + s.{struct_field} = nil + }} else {{ + s.{struct_field} = {isptr}tmp.Value + }}''' + else: + switch_case_format = ''' case {name}: value := {case_type}{{}} if err := json.Unmarshal(data, &value); err != nil {{@@ -374,12 +440,17 @@ def generate_marshal_methods(type: str, case_type = type_dict[name] isptr = "*" if case_type[0] not in "*[" else "" switch_cases += switch_case_format.format(name = name, + struct_field = struct_field, + json_field = json_field, + isptr = isptr, case_type = case_type) if case_type not in added: if_supported_types += f'''typestr != "{case_type}" &&\n\t\t''' added[case_type] = True - marshalfn = f''' + marshalfn = "" + if type_is_union: + marshalfn = f''' func (s {type}) MarshalJSON() ([]byte, error) {{ base, err := json.Marshal(s.{base}) if err != nil {{@@ -564,4 +635,9 @@ def qapi_to_go_type_name(name: str, meta: str) -> str: words = [word for word in name.replace("_", "-").split("-")] name = words[0].title() if words[0].islower() or words[0].isupper() else words[0] name += ''.join(word.title() for word in words[1:]) + + if meta == "event": + name = name[:-3] if name.endswith("Arg") else name + name += meta.title() + return nameSignature · 1 line
2.35.1
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Re: [RFC PATCH v1 5/8] qapi: golang: Generate qapi's event types in Go
Hi,
Quoted 84 lines · Daniel P. Berrangé
Quoted 13 lines · Victor Toso
This patch handles QAPI event types and generates data structures in Go that handles it.
At the moment of this writing, it generates 51 structures (49 events)
In Golang, each event is handled as a Go structure and there is no big difference, in the Go generated code, between what is a QAPI event type and what is a QAPI struct.
Each QAPI event has the suffix 'Event' in its Golang data structure and contains the fields, mandatory and optional, that can be sent or received.
Quoted 5 lines
In addition, there are two structures added to handle QAPI specification for event types: 'Event' and 'EventBase'.
'EventBase' contains @Name and @Timestamp fields and then 'Event' extends 'EventBase' with an @Arg field of type 'Any'.
Again, I don't think we should need to use an Any type here.
Rather than
type EventBase struct { Name string `json:"event"` Timestamp struct { Seconds int64 `json:"seconds"` Microseconds int64 `json:"microseconds"` } `json:"timestamp"` } type Event struct { EventBase Arg Any `json:"data,omitempty"` } type ShutdownEvent struct { Guest bool `json:"guest"` Reason ShutdownCause `json:"reason"` }I think we should just embed EventBase directly in each specific event eg
type Event struct { Name string `json:"event"` Timestamp struct { Seconds int64 `json:"seconds"` Microseconds int64 `json:"microseconds"` } `json:"timestamp"` } type ShutdownEvent struct { Event Event Guest bool `json:"guest"` Reason ShutdownCause `json:"reason"` }Or perhaps better still, use an interface
type Event interface { GetName() string GetTimestamp() string } type Timestamp struct { Seconds int64 `json:"seconds"` Microseconds int64 `json:"microseconds"` } type ShutdownEvent struct { Timestamp Timestamp `json:"timestamp"` Guest bool `json:"guest"` Reason ShutdownCause `json:"reason"` } func (ev *ShutdownEvent) GetName() string { return "SHUTDOWN" }
Quoted 4 lines
That way you can define public APIs taking 'Event' as a type, and impls of the events can be passed directly in/out.
Similar comment for the Command type.
Yeah, if we are removing Any for Unions and Alternates, we can sure do the same for Events and Commands.
Quoted 178 lines
Quoted 178 lines
The 'Event' type implements the Unmarshaler to decode the QMP JSON Object into the correct Golang (event) struct. The goal here is to facilitate receiving Events.
A TODO for this type is to implement Marshaler for 'Event'. It'll containt runtime checks to validate before transforming the struct into a JSON Object.
Example:
```go qmpMsg := `{ "event" : "MIGRATION", "timestamp":{ "seconds": 1432121972, "microseconds": 744001 }, "data":{ "status": "completed" } }` e := Event{} _ = json.Unmarshal([]byte(qmpMsg), &e) // e.Name == "MIGRATION" // e.Arg.(MigrationEvent).Status == MigrationStatusCompleted ```Signed-off-by Victor Toso <[email protected]>--- scripts/qapi/golang.py | 92 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 84 insertions(+), 8 deletions(-)scripts/qapi/golang.py +84
-8@@ -31,9 +31,10 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, prefix: str): super().__init__() - self.target = {name: "" for name in ["alternate", "enum", "helper", "struct", "union"]} + self.target = {name: "" for name in ["alternate", "enum", "event", "helper", "struct", "union"]} self.objects_seen = {} self.schema = None + self.events = {} self._docmap = {} self.golang_package_name = "qapi"@@ -57,6 +58,24 @@ def visit_begin(self, schema): def visit_end(self): self.schema = None + # EventBase and Event are not specified in the QAPI schema, + # so we need to generate it ourselves. + self.target["event"] += ''' +type EventBase struct { + Name string `json:"event"` + Timestamp struct { + Seconds int64 `json:"seconds"` + Microseconds int64 `json:"microseconds"` + } `json:"timestamp"` +} + +type Event struct { + EventBase + Arg Any `json:"data,omitempty"` +} +''' + self.target["event"] += generate_marshal_methods('Event', self.events) + self.target["helper"] += ''' // Creates a decoder that errors on unknown Fields // Returns true if successfully decoded @from string @into type@@ -279,7 +298,28 @@ def visit_command(self, pass def visit_event(self, name, info, ifcond, features, arg_type, boxed): - pass + assert name == info.defn_name + type_name = qapi_to_go_type_name(name, info.defn_meta) + self.events[name] = type_name + + doc = self._docmap.get(name, None) + self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False + content = "" + if self_contained: + doc_struct, _ = qapi_to_golang_struct_docs(doc) + content = generate_struct_type(type_name, "", doc_struct) + else: + assert isinstance(arg_type, QAPISchemaObjectType) + content = qapi_to_golang_struct(name, + doc, + arg_type.info, + arg_type.ifcond, + arg_type.features, + arg_type.base, + arg_type.members, + arg_type.variants) + + self.target["event"] += content def write(self, output_dir: str) -> None: for module_name, content in self.target.items():@@ -351,15 +391,41 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str: }} ''' -# Marshal methods for Union types +# Marshal methods for Event and Union types def generate_marshal_methods(type: str, type_dict: Dict[str, str], discriminator: str = "", base: str = "") -> str: - assert base != "" - discriminator = "base." + discriminator - - switch_case_format = ''' + type_is_union = False + json_field = "" + struct_field = "" + if type == "Event": + base = type + "Base" + discriminator = "base.Name" + struct_field = "Arg" + json_field = "data" + else: + assert base != "" + discriminator = "base." + discriminator + type_is_union = True + + switch_case_format = "" + if not type_is_union: + switch_case_format = ''' + case "{name}": + tmp := struct {{ + Value {isptr}{case_type} `json:"{json_field},omitempty"` + }}{{}} + if err := json.Unmarshal(data, &tmp); err != nil {{ + return err + }} + if tmp.Value == nil {{ + s.{struct_field} = nil + }} else {{ + s.{struct_field} = {isptr}tmp.Value + }}''' + else: + switch_case_format = ''' case {name}: value := {case_type}{{}} if err := json.Unmarshal(data, &value); err != nil {{@@ -374,12 +440,17 @@ def generate_marshal_methods(type: str, case_type = type_dict[name] isptr = "*" if case_type[0] not in "*[" else "" switch_cases += switch_case_format.format(name = name, + struct_field = struct_field, + json_field = json_field, + isptr = isptr, case_type = case_type) if case_type not in added: if_supported_types += f'''typestr != "{case_type}" &&\n\t\t''' added[case_type] = True - marshalfn = f''' + marshalfn = "" + if type_is_union: + marshalfn = f''' func (s {type}) MarshalJSON() ([]byte, error) {{ base, err := json.Marshal(s.{base}) if err != nil {{@@ -564,4 +635,9 @@ def qapi_to_go_type_name(name: str, meta: str) -> str: words = [word for word in name.replace("_", "-").split("-")] name = words[0].title() if words[0].islower() or words[0].isupper() else words[0] name += ''.join(word.title() for word in words[1:]) + + if meta == "event": + name = name[:-3] if name.endswith("Arg") else name + name += meta.title() +
Quoted 12 lines
Quoted 5 lines
scripts/qapi/golang.py +0
-0return nameSignature · 1 line
2.35.1With regards,
DanielSignature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Cheers,
Victor
RFC PATCH 6/8 qapi: golang: Generate qapi's command types in Go
In reply to Victor TosoThis patch handles QAPI command types and generates data structures in Go that decodes from QMP JSON Object to Go data structure and vice versa.
At the time of this writing, it generates 210 structures (208 commands)
This is very similar to previous commit, that handles QAPI union types in Go.
Each QAPI command will generate a Go struct with the suffix 'Command'. Its fields, if any, are the mandatory or optional arguments defined in the QAPI command.
Simlar to Event, this patch adds two structures to handle QAPI specification for command types: 'Command' and 'CommandBase'.
'CommandBase' contains @Id and @Name. 'Command' extends 'CommandBase' with an @Arg field of type 'Any'.
The 'Command' type implements the Unmarshaler to decode QMP JSON Object into the correct Golang (command) struct.
Marshal example:
```go
cmdArg := SetPasswordCommand{}
cmdArg.Protocol = DisplayProtocolVnc
cmdArg.Password = "secret"
cmd := Command{}
cmd.Name = "set_password"
cmd.Arg = cmdArg
b, _ := json.Marshal(&cmd)
// string(b) == `{"execute":"set_password","arguments":{"protocol":"vnc","password":"secret"}}`
```Unmarshal example:
```go
qmpCommand := `
{
"execute": "set_password",
"arguments":{
"protocol": "vnc",
"password": "secret"
}
}`
cmd := Command{}
_ = json.Unmarshal([]byte(qmpCommand), &cmd)
// cmd.Name == "set_password"
// cmd1.Arg.(SetPasswordCommand).Protocol == DisplayProtocolVnc
```--- scripts/qapi/golang.py | 49 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 45 insertions(+), 4 deletions(-)
scripts/qapi/golang.py +45 -4
@@ -31,10 +31,11 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, prefix: str): super().__init__() - self.target = {name: "" for name in ["alternate", "enum", "event", "helper", "struct", "union"]} + self.target = {name: "" for name in ["alternate", "command", "enum", "event", "helper", "struct", "union"]} self.objects_seen = {} self.schema = None self.events = {} + self.commands = {} self._docmap = {} self.golang_package_name = "qapi"
@@ -76,6 +77,19 @@ def visit_end(self): ''' self.target["event"] += generate_marshal_methods('Event', self.events) + self.target["command"] += ''' +type CommandBase struct { + Id string `json:"id,omitempty"` + Name string `json:"execute"` +} + +type Command struct { + CommandBase + Arg Any `json:"arguments,omitempty"` +} +''' + self.target["command"] += generate_marshal_methods('Command', self.commands) + self.target["helper"] += ''' // Creates a decoder that errors on unknown Fields // Returns true if successfully decoded @from string @into type
@@ -295,7 +309,29 @@ def visit_command(self, allow_oob: bool, allow_preconfig: bool, coroutine: bool) -> None: - pass + assert name == info.defn_name + type_name = qapi_to_go_type_name(name, info.defn_meta) + self.commands[name] = type_name + + doc = self._docmap.get(name, None) + self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False + content = "" + if boxed or self_contained: + args = "" if not arg_type else "\n" + arg_type.name + doc_struct, _ = qapi_to_golang_struct_docs(doc) + content = generate_struct_type(type_name, args, doc_struct) + else: + assert isinstance(arg_type, QAPISchemaObjectType) + content = qapi_to_golang_struct(name, + doc, + arg_type.info, + arg_type.ifcond, + arg_type.features, + arg_type.base, + arg_type.members, + arg_type.variants) + + self.target["command"] += content def visit_event(self, name, info, ifcond, features, arg_type, boxed): assert name == info.defn_name
@@ -391,7 +427,7 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str: }} ''' -# Marshal methods for Event and Union types +# Marshal methods for Event, Commad and Union types def generate_marshal_methods(type: str, type_dict: Dict[str, str], discriminator: str = "",
@@ -404,6 +440,11 @@ def generate_marshal_methods(type: str, discriminator = "base.Name" struct_field = "Arg" json_field = "data" + elif type == "Command": + base = type + "Base" + discriminator = "base.Name" + struct_field = "Arg" + json_field = "arguments" else: assert base != "" discriminator = "base." + discriminator
@@ -636,7 +677,7 @@ def qapi_to_go_type_name(name: str, meta: str) -> str: name = words[0].title() if words[0].islower() or words[0].isupper() else words[0] name += ''.join(word.title() for word in words[1:]) - if meta == "event": + if meta == "event" or meta == "command": name = name[:-3] if name.endswith("Arg") else name name += meta.title()
Signature · 1 line
2.35.1
RFC PATCH 7/8 qapi: golang: Add CommandResult type to Go
In reply to Victor TosoThis patch adds a struct type in Go that will handle return values for QAPI's command types.
The return value of a Command is, encouraged to be, QAPI's complext types or an Array of those.
The 'CommandResult' type acts in similar fashion to 'Event' and 'Command', in order to map the right return data structure based on the issued 'Command'.
This patch also adds a syntax sugar method to 'Command' to return the 'CommandResult' struct to use when receiving the return data.
Example:
```go
cmd := Command{}
cmd.Name = `query-tpm-models`
// bytes, _ := json.Marshal(&cmd)
// send bytes ...
received := `{"return":["tpm-tis","tpm-crb","tpm-spapr"]}`
cmdRet := cmd.GetReturnType()
_ = json.Unmarshal([]byte(received), &cmdRet)
// cmdRet.Value.([]TpmModel)[2] == TpmModelTpmSpapr
```--- scripts/qapi/golang.py | 42 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 40 insertions(+), 2 deletions(-)
scripts/qapi/golang.py +40 -2
@@ -36,6 +36,7 @@ def __init__(self, prefix: str): self.schema = None self.events = {} self.commands = {} + self.command_results = {} self._docmap = {} self.golang_package_name = "qapi"
@@ -90,6 +91,32 @@ def visit_end(self): ''' self.target["command"] += generate_marshal_methods('Command', self.commands) + self.target["command"] += ''' +type CommandResult struct { + CommandBase + Value Any `json:"return,omitempty"` +} + +func (s Command) GetReturnType() CommandResult { + return CommandResult{ + CommandBase: s.CommandBase, + } +} + +// In order to evaluate nil value to empty JSON object +func (s *CommandResult) MarshalJSON() ([]byte, error) { + if s.Value == nil { + return []byte(`{"return":{}}`), nil + } + tmp := struct { + Value Any `json:"return"` + }{Value: s.Value} + + return json.Marshal(&tmp) +} +''' + self.target["command"] += generate_marshal_methods('CommandResult', self.command_results) + self.target["helper"] += ''' // Creates a decoder that errors on unknown Fields // Returns true if successfully decoded @from string @into type
@@ -312,6 +339,9 @@ def visit_command(self, assert name == info.defn_name type_name = qapi_to_go_type_name(name, info.defn_meta) self.commands[name] = type_name + if ret_type: + ret_type_name = qapi_schema_type_to_go_type(ret_type.name) + self.command_results[name] = ret_type_name doc = self._docmap.get(name, None) self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False
@@ -445,6 +475,11 @@ def generate_marshal_methods(type: str, discriminator = "base.Name" struct_field = "Arg" json_field = "arguments" + elif type == "CommandResult": + base = "CommandBase" + discriminator = "s.Name" + struct_field = "Value" + json_field = "return" else: assert base != "" discriminator = "base." + discriminator
@@ -527,14 +562,17 @@ def generate_marshal_methods(type: str, return []byte(result), nil }} ''' - unmarshal_base = f''' + unmarshal_base = "" + unmarshal_default_warn = "" + if type != "CommandResult": + unmarshal_base = f''' var base {base} if err := json.Unmarshal(data, &base); err != nil {{ return err }} s.{base} = base ''' - unmarshal_default_warn = f''' + unmarshal_default_warn = f''' default: fmt.Println("Failed to decode {type}", {discriminator})'''
Signature · 1 line
2.35.1
RFC PATCH 8/8 qapi: golang: document skip function visit_array_types
In reply to Victor Toso--- scripts/qapi/golang.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
scripts/qapi/golang.py +6 -1
@@ -321,7 +321,12 @@ def visit_enum_type(self: QAPISchemaGenGolangVisitor, ''' def visit_array_type(self, name, info, ifcond, element_type): - pass + # TLDR: We don't need to any extra boilerplate in Go to handle Arrays. + # + # This function is implemented just to be sure that: + # 1. Every array type ends with List + # 2. Every array type's element is the array type without 'List' + assert name.endswith("List") and name[:-4] == element_type.name def visit_command(self, name: str,
Signature · 1 line
2.35.1
Quoted 2 lines · Victor Toso
Thanks for taking a look, let me know if you have questions, ideas or suggestions.
Full disclosure: I have only given the actual implementation a very cursory look so far, and I've focused on the generated Go API instead.
Overall things look pretty good.
One concern that I have is about naming struct members: things like SpiceInfo.MouseMode and most others are translated from the QAPI schema exactly the way you'd expect them, but for example ChardevCommon.Logappend doesn't look quite right. Of course there's no way to programmatically figure out what to capitalize, but maybe there's room for adding this kind of information in the form of additional annotations or something like that? Same for the various structs or members that have unexpectedly-capitalized "Tls" or "Vnc" in them.
To be clear, I don't think the above is a blocker - just something to be aware of, and think about.
My biggest concern is about the interface offered for commands.
Based on the example you have in the README and how commands are defined, invoking (a simplified version of) the trace-event-get-state command would look like
cmd := Command{
Name: "trace-event-get-state",
Arg: TraceEventGetStateCommand{
Name: "qemu_memalign",
},
}
qmp_input, _ := json.Marshal(&cmd)
// qmp_input now contains
// {"execute":"trace-event-get-state","arguments":{"name":"qemu_memalign"}}
// do something with it qmp_output :=
([]byte)({"return":{"name":"qemu_memalign","state":"disabled"}})
ret := cmd.GetReturnType()
_ = json.Unmarshal(qmp_output, &ret)
// ret is a CommandResult instance whose Value member can be cast // to a TraceEventInfo struct
First of all, from an application's point of view there are way too many steps involved: performing this operation should really be as simple as
ret, _ := qmp.TraceEventGetState("qemu_memalign") // ret is a TraceEventInfo instance
That's the end state we should be working towards.
Of course that assumes that the "qmp" object knows where the QMP socket is, knows how to talk the QMP protocol, transparently deals with serializing and deserializing data... Plus, in some case you might want to deal with the wire transfer yourself in an application-specific manner. So it makes sense to have the basic building blocks available and then build the more ergonomic SDK on top of that - with only the first part being in scope for this series.
Even with that in mind, the current interface is IMO problematic because of its almost complete lack of type safety. Both Command.Arg and CommandResult.Value are of type Any and CommandBase.Name, which is used to drive the JSON unmarshal logic as well as ending up on the wire when executing a command, is just a plain string.
I think the low-level interface should look more like
cmd := TraceEventGetStateCommand{
Name: "qemu_memalign",
}
qmp_input, _ := json.Marshal(&cmd)
// qmp_input looks the same as before qmp_output :=
([]byte)({"return":{"name":"qemu_memalign","state":"disabled"}})
ret := TraceEventInfo{}
_ = json.Unmarshal(qmp_output, &ret)
// ret is a TraceEventInfo instance
The advantages over the current implementation is that the compiler will prevent you from doing something silly like passing the wrong set of arguments to a commmand, and that the application has to explicitly spell out what kind of object it expects to get as output.
I'm attaching an incomplete implementation that I used for playing around. It's obviously too simplistic, but hopefully it will help illustrate my point.
Dealing with errors and commands that don't have a return value might require us to have generic CommandResult wrapper after all, but we should really try as hard as we can to stick to type safe interfaces.
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
package main
import (
"encoding/json" "fmt"
)
type TraceEventGetStateCommand struct {
Name string json:"name"
}
func (self *TraceEventGetStateCommand) MarshalJSON() ([]byte, error) {
type Arguments TraceEventGetStateCommand
return json.Marshal(&struct {
Execute string `json:"execute"`
Arguments *Arguments `json:"arguments"`
}{
Execute: "trace-event-get-state",
Arguments: (*Arguments)(self),
})
}type TraceEventInfo struct {
Name string json:"name"
State string json:"state"
}
func (self *TraceEventInfo) UnmarshalJSON(data []byte) error {
type Return TraceEventInfo
ret := struct {
Return Return json:"return"
}{}
err := json.Unmarshal(data, &ret)
if err != nil {
return err
}
self.Name = ret.Return.Name
self.State = ret.Return.State
return nil
}
func main() {
var err error
var qmp_input []byte
var qmp_output []byte
cmd := TraceEventGetStateCommand{
Name: "qemu_memalign",
}
if qmp_input, err = json.Marshal(&cmd); err != nil {
panic(err)
}
fmt.Printf(" cmd: %v\n", cmd)
fmt.Printf("-> qmp_input: %v\n", string(qmp_input))
qmp_output = ([]byte)(`{"return":{"name":"qemu_memalign","state":"disabled"}}`)
ret := TraceEventInfo{}
if err = json.Unmarshal(qmp_output, &ret); err != nil {
panic(err)
}
fmt.Printf("<- qmp_output: %v\n", string(qmp_output))
fmt.Printf(" ret: %v\n", ret)
}Quoted 3 lines · Andrea Bolognani
Dealing with errors and commands that don't have a return value might require us to have generic CommandResult wrapper after all, but we should really try as hard as we can to stick to type safe interfaces.
On second thought, this wouldn't actually need to be generic: we could have something like
type TraceEventGetStateResult struct {
Result TraceEventInfo `json:"return"`
Error *Error `json:"error"`
}and the caller would check that res.Error is nil before accessing res.Result.
Commands for which a return value is not expected would just have the Error part in their corresponding Result struct, and those that can either return an object or nothing (are there actually any like that?) could have a pointer as the Result member.
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Quoted 8 lines · Andrea Bolognani
Quoted 2 lines · Victor Toso
Thanks for taking a look, let me know if you have questions, ideas or suggestions.
Full disclosure: I have only given the actual implementation a very cursory look so far, and I've focused on the generated Go API instead.
Quoted 6 lines
Overall things look pretty good.
One concern that I have is about naming struct members: things like SpiceInfo.MouseMode and most others are translated from the QAPI schema exactly the way you'd expect them, but for example ChardevCommon.Logappend doesn't look quite right.
It doesn't look quite right in the QAPI schema, either: @logappend. If it was @log-append, as it should, then it would get translated to LogAppend, I guess.
Fixing up style isn't a code generator's job.
Quoted 2 lines
Of course there's
no way to programmatically figure out what to capitalize,
Some case conversions are straightforward enough. For instance, the C code generator generates qapi_event_send_some_event() for event SOME_EVENT, and inclusion guard macro FILE_NAME_H for module file name file-name.json. No magic involved.
Conversion from lower-case-with-dashes to CamelCase doesn't have to be magic, either. You just have to accept garbage-in (like missing dashes) will give you garbage-out. I wouldn't care too much; CamelCaseIsAnIllegibleMessAnyway.
Conversion from CamelCase is always trouble, though. There is one instance so far: generating C enumeration constants. We want TYPE_NAME
+ '_' + MEMBER_NAME, where TYPE_NAME is the enumeration type's name converted from CamelCase to UPPER_CASE_WITH_UNDERSCORES, and MEMBER_NAME is the member name converted from lower-case-with-dashes to UPPER_CASE_WITH_UNDERSCORES.
camel_to_upper() tries, but the result is often unappealing, surprising, or both.
Quoted 3 lines
but maybe
there's room for adding this kind of information in the form of additional annotations or something like that?
We did for enumeration types: 'prefix' overrides the TYPE_NAME prefix. I fear this was a mistake.
Quoted 3 lines
Same for the various
structs or members that have unexpectedly-capitalized "Tls" or "Vnc" in them.
Examples?
Quoted 2 lines
To be clear, I don't think the above is a blocker - just something to be aware of, and think about.
Yup.
[...]
Quoted 11 lines · Markus Armbruster
Quoted 4 lines · Andrea Bolognani
One concern that I have is about naming struct members: things like SpiceInfo.MouseMode and most others are translated from the QAPI schema exactly the way you'd expect them, but for example ChardevCommon.Logappend doesn't look quite right.
It doesn't look quite right in the QAPI schema, either: @logappend. If it was @log-append, as it should, then it would get translated to LogAppend, I guess.
Fixing up style isn't a code generator's job.
I agree that the generator shouldn't take too many liberties when translating names, and specifically should never attempt to figure out that @logappend should have been @log-append instead.
What I was thinking of was more along the lines of, can we change the schema so that the proper name is available to the generator without breaking the wire protocol? Maybe something like
##
# ChardevCommon:
#
# @logappend (rename @log-append): ...
##
That way the generator would have access to both information, and would thus be able to generate
type ChardevCommon struct {
LogAppend *bool `json:"logappend,omitempty"`
}The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.
Quoted 5 lines
Quoted 3 lines
Same for the various
structs or members that have unexpectedly-capitalized "Tls" or "Vnc" in them.Examples?
A perfect one is TlsCredsProperties, whose endpoint member is of type QCryptoTLSCredsEndpoint.
On the VNC front, we have SetPasswordOptionsVnc but also DisplayReloadOptionsVNC.
There's plenty more, but this should be illustrative enough already. Capitalization of these acronyms is inconsistent across the schema, with one of the two forms disagreeing with Go naming expectations.
In this case we might be able to just change the schema without introducing backwards compatibility issues, though? Type names are not actually transmitted on the wire IIUC.
Quoted 6 lines
Quoted 3 lines
but maybe
there's room for adding this kind of information in the form of additional annotations or something like that?We did for enumeration types: 'prefix' overrides the
TYPE_NAMEprefix. I fear this was a mistake.
This might be an oversimplification, but I think we might be able to solve all of these issues with a single annotation in the form
namespace:word-MLA-other-words
So for example QCryptoTLSCredsEndpoint would be annotated with
q:crypto-TLS-creds-endpoint
and each generator would have enough information to produce identifiers that fit into the corresponding language, such as
qcrypto_tls_creds_endpoint
CryptoTlsCredsEndpoint
or whatever else.
Of course such annotations would only be necessary to deal with identifiers that are not already following the expected naming conventions and when MLAs are involved.
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Quoted 30 lines · Andrea Bolognani
Quoted 11 lines · Markus Armbruster
Quoted 4 lines · Andrea Bolognani
One concern that I have is about naming struct members: things like SpiceInfo.MouseMode and most others are translated from the QAPI schema exactly the way you'd expect them, but for example ChardevCommon.Logappend doesn't look quite right.
It doesn't look quite right in the QAPI schema, either: @logappend. If it was @log-append, as it should, then it would get translated to LogAppend, I guess.
Fixing up style isn't a code generator's job.
I agree that the generator shouldn't take too many liberties when translating names, and specifically should never attempt to figure out that @logappend should have been @log-append instead.
What I was thinking of was more along the lines of, can we change the schema so that the proper name is available to the generator without breaking the wire protocol? Maybe something like
##
# ChardevCommon:
#
# @logappend (rename @log-append): ...
##That way the generator would have access to both information, and would thus be able to generate
Quoted 6 lines
type ChardevCommon struct { LogAppend *bool `json:"logappend,omitempty"` }The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.
At the price of mild inconsistency between the library interface and QMP.
We could clean up QMP if we care, keeping around the old names for compatibility. See also Kevin's work on QAPI aliases. Which is much more ambitious, though.
Quoted 6 lines
Quoted 5 lines
Quoted 3 lines
Same for the various
structs or members that have unexpectedly-capitalized "Tls" or "Vnc" in them.Examples?
Quoted 8 lines
A perfect one is TlsCredsProperties, whose endpoint member is of type QCryptoTLSCredsEndpoint.
On the VNC front, we have SetPasswordOptionsVnc but also DisplayReloadOptionsVNC.
There's plenty more, but this should be illustrative enough already. Capitalization of these acronyms is inconsistent across the schema,
Common issue with camel-cased compounds containing acronyms, because either way is ugly.
Quoted 1 line
with one of the two forms disagreeing with Go naming expectations.
Pardon my ignorance: What are Go's expectations?
Quoted 3 lines
In this case we might be able to just change the schema without introducing backwards compatibility issues, though? Type names are not actually transmitted on the wire IIUC.
Correct!
Quoted 19 lines
Quoted 6 lines
Quoted 3 lines
but maybe
there's room for adding this kind of information in the form of additional annotations or something like that?We did for enumeration types: 'prefix' overrides the
TYPE_NAMEprefix. I fear this was a mistake.This might be an oversimplification, but I think we might be able to solve all of these issues with a single annotation in the form
namespace:word-MLA-other-words
So for example QCryptoTLSCredsEndpoint would be annotated with
q:crypto-TLS-creds-endpoint
and each generator would have enough information to produce identifiers that fit into the corresponding language, such as
Quoted 8 lines
qcrypto_tls_creds_endpoint
CryptoTlsCredsEndpointor whatever else.
Of course such annotations would only be necessary to deal with identifiers that are not already following the expected naming conventions and when MLAs are involved.
Pardon my ignorance some more: what are MLAs?
Quoted 6 lines · Markus Armbruster
Quoted 2 lines · Andrea Bolognani
The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.
At the price of mild inconsistency between the library interface and QMP.
That's fine, and in fact it already happens all the time when QAPI names (log-append) are translated to C identifiers (log_append).
Quoted 3 lines
We could clean up QMP if we care, keeping around the old names for compatibility. See also Kevin's work on QAPI aliases. Which is much more ambitious, though.
I wasn't aware of that effort. Personally I'm always in favor of cleaning up inconsistencies, so I am automatically a fan :)
That said, the idea of exposing a sub-par Go API until such massive undertaking can be completed is not terribly appealing. And it would not address every facet of the issue (see below).
Quoted 4 lines
Quoted 1 line
Capitalization of these acronyms is inconsistent across the schema,
Common issue with camel-cased compounds containing acronyms, because either way is ugly.
Agreed :) But consistent ugliness is still preferable to inconsistent ugliness.
Quoted 3 lines
Quoted 1 line
with one of the two forms disagreeing with Go naming expectations.
Pardon my ignorance: What are Go's expectations?
Acronyms are usually all upper case:
https://pkg.go.dev/net/http#ParseHTTPVersion
https://pkg.go.dev/net/http#ProxyURL
https://pkg.go.dev/crypto/tls#NewLRUClientSessionCache
The same seems to be true of Python:
https://docs.python.org/3/library/http.html#http.HTTPStatus
https://docs.python.org/3/library/urllib.error.html#urllib.error.URLError
https://docs.python.org/3/library/xmlrpc.server.html#xmlrpc.server.SimpleXMLRPCServer
Rust, on the other hand, seems to prefer only capitalizing the first letter of a word, no matter if it's an acronym:
https://doc.rust-lang.org/std/net/struct.TcpStream.html
https://doc.rust-lang.org/std/net/struct.Ipv4Addr.html
https://doc.rust-lang.org/std/ffi/struct.OsString.html
Whether different naming conventions are used for types, functions and struct members is also language-dependent.
Quoted 5 lines
Quoted 3 lines
In this case we might be able to just change the schema without introducing backwards compatibility issues, though? Type names are not actually transmitted on the wire IIUC.
Correct!
That's great, but even if we decided to change all type names so that the schema is internally consistent and follows a naming convention that's reasonable for C, Go and Python, we'd still leave the Rust interface looking weird... There's no one-size-fits-all name, unfortunately.
Quoted 5 lines
Quoted 3 lines
Of course such annotations would only be necessary to deal with identifiers that are not already following the expected naming conventions and when MLAs are involved.
Pardon my ignorance some more: what are MLAs?
Multi Letter Acronyms. Which are actually just called "acronyms" I guess? O:-)
The point is that, if we want to provide a language interface that feels natural, we need a way to mark parts of a QAPI symbol's name in a way that makes it possible for the generator to know they're acronyms and treat them in an appropriate, language-specific manner.
The obvious way to implement this would be with an annotation along the lines of the one I proposed. Other ideas?
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Quoted 10 lines · Andrea Bolognani
Quoted 6 lines · Markus Armbruster
Quoted 2 lines · Andrea Bolognani
The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.
At the price of mild inconsistency between the library interface and QMP.
That's fine, and in fact it already happens all the time when QAPI names (log-append) are translated to C identifiers (
log_append).
There's a difference between trivial translations like "replace '-' by '_'" and arbitrary replacement like the one for enumeration constants involving 'prefix'.
Quoted 9 lines
Quoted 3 lines
We could clean up QMP if we care, keeping around the old names for compatibility. See also Kevin's work on QAPI aliases. Which is much more ambitious, though.
I wasn't aware of that effort. Personally I'm always in favor of cleaning up inconsistencies, so I am automatically a fan :)
That said, the idea of exposing a sub-par Go API until such massive undertaking can be completed is not terribly appealing.
Point.
Quoted 10 lines
And it would
not address every facet of the issue (see below).Quoted 4 lines
Quoted 1 line
Capitalization of these acronyms is inconsistent across the schema,
Common issue with camel-cased compounds containing acronyms, because either way is ugly.
Agreed :) But consistent ugliness is still preferable to inconsistent ugliness.
True.
Quoted 16 lines
Quoted 3 lines
Quoted 1 line
with one of the two forms disagreeing with Go naming expectations.
Pardon my ignorance: What are Go's expectations?
Acronyms are usually all upper case:
https://pkg.go.dev/net/http#ParseHTTPVersion
https://pkg.go.dev/net/http#ProxyURL
https://pkg.go.dev/crypto/tls#NewLRUClientSessionCacheThe same seems to be true of Python:
https://docs.python.org/3/library/http.html#http.HTTPStatus
https://docs.python.org/3/library/urllib.error.html#urllib.error.URLError
https://docs.python.org/3/library/xmlrpc.server.html#xmlrpc.server.SimpleXMLRPCServer
Quoted 6 lines
Rust, on the other hand, seems to prefer only capitalizing the first letter of a word, no matter if it's an acronym:
https://doc.rust-lang.org/std/net/struct.TcpStream.html
https://doc.rust-lang.org/std/net/struct.Ipv4Addr.html
https://doc.rust-lang.org/std/ffi/struct.OsString.html
Another strange game where the only winning move is not to play.
Quoted 2 lines
Whether different naming conventions are used for types, functions and struct members is also language-dependent.
Yes.
Quoted 11 lines
Quoted 5 lines
Quoted 3 lines
In this case we might be able to just change the schema without introducing backwards compatibility issues, though? Type names are not actually transmitted on the wire IIUC.
Correct!
That's great, but even if we decided to change all type names so that the schema is internally consistent and follows a naming convention that's reasonable for C, Go and Python, we'd still leave the Rust interface looking weird... There's no one-size-fits-all name, unfortunately.
C will remain the primary customer for quite a while. It doesn't come with universally accepted naming conventions, so we made up our own. I think we have some wiggle room there. We could, for instance, decide to clean up the current inconsistent capitalization of acronyms in the QAPI schema to either style, TCPStream or TcpStream.
Your point remains: some names will still look "weird" in some possible target languages.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Of course such annotations would only be necessary to deal with identifiers that are not already following the expected naming conventions and when MLAs are involved.
Pardon my ignorance some more: what are MLAs?
Multi Letter Acronyms. Which are actually just called "acronyms" I guess? O:-)
Well played!
Quoted 4 lines
The point is that, if we want to provide a language interface that feels natural, we need a way to mark parts of a QAPI symbol's name in a way that makes it possible for the generator to know they're acronyms and treat them in an appropriate, language-specific manner.
It's not just acronyms. Consider IAmALittleTeapot. If you can assume that only beginning of words are capitalized, even for acronyms, you can split this into words without trouble. You can't recover correct case,
though: "i am a little teapot" is wrong.
"Split before capital letter" falls apart when you have characters that cannot be capitalized: Point3d.
Camel case is hopeless.
Quoted 2 lines
The obvious way to implement this would be with an annotation along the lines of the one I proposed. Other ideas?
I'm afraid having the schema spell out names in multiple naming conventions could be onerous. How many names will need it? Times how many naming conventions?
Another issue: the fancier the translation from schema name to language-specific name gets, the harder it becomes to find one from the other.
Quoted 7 lines · Markus Armbruster
Quoted 6 lines · Andrea Bolognani
Quoted 5 lines
Quoted 2 lines
The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.
At the price of mild inconsistency between the library interface and QMP.
Quoted 6 lines
Quoted 2 lines
That's fine, and in fact it already happens all the time when QAPI names (log-append) are translated to C identifiers (
log_append).There's a difference between trivial translations like "replace '-' by '_'" and arbitrary replacement like the one for enumeration constants involving 'prefix'.
Fair enough.
I still feel that 1) users of a language SDK will ideally not need to look at the QAPI schema or wire chatter too often and 2) even when that ends up being necessary, figuring out that LogAppend and logappend are the same thing is not going to be an unreasonable hurdle, especially when the status quo would be to work with Logappend instead.
Quoted 9 lines
Quoted 4 lines
The point is that, if we want to provide a language interface that feels natural, we need a way to mark parts of a QAPI symbol's name in a way that makes it possible for the generator to know they're acronyms and treat them in an appropriate, language-specific manner.
It's not just acronyms. Consider IAmALittleTeapot. If you can assume that only beginning of words are capitalized, even for acronyms, you can split this into words without trouble. You can't recover correct case,
though: "i am a little teapot" is wrong.
Is there any scenario in which we would care though? We're in the business of translating identifiers from one machine representation to another, so once it has been split up correctly into the words that compose it (which in your example above it has) then we don't really care about anything else unless acronyms are involved.
In other words, we can obtain the list of words "i am a little teapot" programmatically both from IAmALittleTeapot and i-am-a-little-teapot, and in both cases we can then generate IAmALittleTeapot or I_AM_A_LITTLE_TEAPOT or i_am_a_little_teapot or whatever is appropriate for the context and target language, but the fact that in a proper English sentence "I" would have to be capitalized doesn't really enter the picture.
Quoted 4 lines
"Split before capital letter" falls apart when you have characters that cannot be capitalized: Point3d.
Camel case is hopeless.
I would argue that it works quite well for most scenarios, but there are some corner cases where it's clearly not good enough. If we can define a way to clue in the generator about "Point3d" having to be interpreted as "point 3d" and "VNCProps" as "vnc props", then we are golden. That wouldn't be necessary for simple cases that are already handled correctly.
A more radical idea would be to start using dash-notation for types too. That'd remove the word splitting issue altogether, at the cost of the schema being (possibly) harder to read and more distanced from the generated code.
You'd still only be able to generate VncProps from vnc-props though.
Quoted 5 lines
Quoted 2 lines
The obvious way to implement this would be with an annotation along the lines of the one I proposed. Other ideas?
I'm afraid having the schema spell out names in multiple naming conventions could be onerous. How many names will need it?
I don't have hard data on this. I could try extracting it, but that might end up being a bigger job than I had anticipated.
My guess is that the number of cases where the naive algorithm can't split words correctly is relatively small compared to the size of the entire QAPI schema. Fair warning: I have made incorrect guesses in the past ;)
Quoted 1 line
Times how many naming conventions?
Yeah, I don't think requiring all possible permutations to be spelled out in the schema is the way to go. That's exactly why my proposal was to offer a way to inject the semantic information that the parser can't figure out itself.
Once you have a way to inform the generator that "VNCProps" is made of the two words "vnc" and "props", and that "vnc" is an acronym, then it can generate an identifier appropriate for the target language without having to spell out anywhere that such an identifier would be VNCProps for Go and VncProps for Rust.
By the way, while looking around I realized that we also have to take into account things like D-Bus: the QAPI type ChardevDBus, for example, would probably translate verbatim to Go but have to be changed to ChardevDbus for Rust. Fun :)
Revised proposal for the annotation:
ns:word-WORD-WoRD-123Word
Words are always separated by dashes; "regular" words are entirely lowercase, while the presence of even a single uppercase letter in a word denotes the fact that its case should be preserved when the naming conventions of the target language allow that.
Quoted 3 lines
Another issue: the fancier the translation from schema name to language-specific name gets, the harder it becomes to find one from the other.
That's true, but at least to me the trade-off feels reasonable.
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Quoted 11 lines · Andrea Bolognani
Quoted 10 lines · Markus Armbruster
Quoted 8 lines · Andrea Bolognani
Quoted 5 lines
Quoted 2 lines
The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.
At the price of mild inconsistency between the library interface and QMP.
That's fine, and in fact it already happens all the time when QAPI names (log-append) are translated to C identifiers (
log_append).
Quoted 8 lines
Quoted 3 lines
There's a difference between trivial translations like "replace '-' by '_'" and arbitrary replacement like the one for enumeration constants involving 'prefix'.
Fair enough.
I still feel that 1) users of a language SDK will ideally not need to look at the QAPI schema or wire chatter too often
I think the most likely point of contact is the QEMU QMP Reference Manual.
Quoted 5 lines
even when
that ends up being necessary, figuring out that LogAppend and logappend are the same thing is not going to be an unreasonable hurdle, especially when the status quo would be to work with Logappend instead.
Yes, it's "mild inconsistency", hardly an unreasonable hurdle. I think it gets in the way mostly when searching documentation. Differences in case are mostly harmless, just use case-insensitive search. Use of '_' vs '-' would also be harmless (just do the replacement), if the use of '-' in the schema was consistent. Sadly, it's not, and that's already a perennial low-level annoyance.
My point is: a name override feature like the one you propose needs to be used with discipline and restraint. Adds to reviewers' mental load. Needs to be worth it. I'm not saying it isn't, I'm just pointing out a cost.
Quoted 16 lines
Quoted 9 lines
Quoted 4 lines
The point is that, if we want to provide a language interface that feels natural, we need a way to mark parts of a QAPI symbol's name in a way that makes it possible for the generator to know they're acronyms and treat them in an appropriate, language-specific manner.
It's not just acronyms. Consider IAmALittleTeapot. If you can assume that only beginning of words are capitalized, even for acronyms, you can split this into words without trouble. You can't recover correct case,
though: "i am a little teapot" is wrong.Is there any scenario in which we would care though? We're in the business of translating identifiers from one machine representation to another, so once it has been split up correctly into the words that compose it (which in your example above it has) then we don't really care about anything else unless acronyms are involved.
Quoted 7 lines
In other words, we can obtain the list of words "i am a little teapot" programmatically both from IAmALittleTeapot and i-am-a-little-teapot, and in both cases we can then generate IAmALittleTeapot or
I_AM_A_LITTLE_TEAPOTori_am_a_little_teapotor whatever is appropriate for the context and target language, but the fact that in a proper English sentence "I" would have to be capitalized doesn't really enter the picture.
My point is that conversion from CamelCase has two sub-problems:
splitting words and recovering case. Splitting words is easy when exactly the beginning of words is capitalized. Recovering case is guesswork. Most English words are all lower case, but some start with a capital letter, and acronyms are all caps.
Wild idea: assume all lower case, but keep a list of exceptions.
Quoted 11 lines
Quoted 4 lines
"Split before capital letter" falls apart when you have characters that cannot be capitalized: Point3d.
Camel case is hopeless.
I would argue that it works quite well for most scenarios, but there are some corner cases where it's clearly not good enough. If we can define a way to clue in the generator about "Point3d" having to be interpreted as "point 3d" and "VNCProps" as "vnc props", then we are golden. That wouldn't be necessary for simple cases that are already handled correctly.
Hyphenization rules? *Cough* *cough*
Quoted 4 lines
A more radical idea would be to start using dash-notation for types too. That'd remove the word splitting issue altogether, at the cost of the schema being (possibly) harder to read and more distanced from the generated code.
Yes.
Quoted 10 lines
You'd still only be able to generate VncProps from vnc-props though.
Quoted 5 lines
Quoted 2 lines
The obvious way to implement this would be with an annotation along the lines of the one I proposed. Other ideas?
I'm afraid having the schema spell out names in multiple naming conventions could be onerous. How many names will need it?
I don't have hard data on this. I could try extracting it, but that might end up being a bigger job than I had anticipated.
I figure extracting is easier for me than for you. But let's have a closer look at the job at hand first.
The QAPI schema language uses three naming styles:
* lower-case-with-hyphens for command and member names
Many names use upper case and '_'. See pragma command-name-exceptions and member-name-exceptions.
Some (many?) names lack separators between words (example: logappend).
* UPPER_CASE_WITH_UNDERSCORE for event names
* CamelCase for type names
Capitalization of words is inconsistent in places (example: VncInfo vs. DisplayReloadOptionsVNC).
What style conversions will we need for Go? Any other conversions come to mind?
What problems do these conversions have?
Quoted 23 lines
My guess is that the number of cases where the naive algorithm can't split words correctly is relatively small compared to the size of the entire QAPI schema. Fair warning: I have made incorrect guesses in the past ;)
Quoted 1 line
Times how many naming conventions?
Yeah, I don't think requiring all possible permutations to be spelled out in the schema is the way to go. That's exactly why my proposal was to offer a way to inject the semantic information that the parser can't figure out itself.
Once you have a way to inform the generator that "VNCProps" is made of the two words "vnc" and "props", and that "vnc" is an acronym, then it can generate an identifier appropriate for the target language without having to spell out anywhere that such an identifier would be VNCProps for Go and VncProps for Rust.
By the way, while looking around I realized that we also have to take into account things like D-Bus: the QAPI type ChardevDBus, for example, would probably translate verbatim to Go but have to be changed to ChardevDbus for Rust. Fun :)
Quoted 8 lines
Revised proposal for the annotation:
ns:word-WORD-WoRD-123Word
Words are always separated by dashes; "regular" words are entirely lowercase, while the presence of even a single uppercase letter in a word denotes the fact that its case should be preserved when the naming conventions of the target language allow that.
Is a word always capitalized the same for a single target language? Or could capitalization depend on context?
Quoted 5 lines
Quoted 3 lines
Another issue: the fancier the translation from schema name to language-specific name gets, the harder it becomes to find one from the other.
That's true, but at least to me the trade-off feels reasonable.
Quoted 6 lines · Markus Armbruster
Quoted 2 lines · Andrea Bolognani
I still feel that 1) users of a language SDK will ideally not need to look at the QAPI schema or wire chatter too often
I think the most likely point of contact is the QEMU QMP Reference Manual.
Note that there isn't anything preventing us from including the original QAPI name in the documentation for the corresponding Go symbol, or even a link to the reference manual.
So we could make jumping from the Go API documentation, which is what a Go programmer will be looking at most of the time, to the QMP documentation pretty much effortless.
Quoted 4 lines
My point is: a name override feature like the one you propose needs to be used with discipline and restraint. Adds to reviewers' mental load. Needs to be worth it. I'm not saying it isn't, I'm just pointing out a cost.
Yeah, I get that.
Note that I'm not suggesting it should be possible for a name to be completely overridden - I just want to make it possible for a human to provide the name parsing algorithm solutions to those problems it can't figure out on its own.
We could prevent that feature from being misused by verifying that the symbol the annotation is attached to can be derived from the list of words provided. That way, something like
# SOMEName (completely-DIFFERENT-name)
would be rejected and we would avoid misuse.
Quoted 1 line
Wild idea: assume all lower case, but keep a list of exceptions.
That could actually work reasonably well for QEMU because we only need to handle correctly what's in the schema, not arbitrary input.
There's always the risk of the list of exceptions getting out of sync with the needs of the schema, but there's similarly no guarantee that annotations are going to be introduced when they are necessary, so it's mostly a wash.
The only slight advantage of the annotation approach would be that it might be easier to notice it being missing because it's close to the name it refers to, while the list of exceptions is tucked away in a script far away from it.
Quoted 6 lines
The QAPI schema language uses three naming styles:
* lower-case-with-hyphens for command and member names
Many names use upper case and '_'. See pragma command-name-exceptions and member-name-exceptions.
Looking at the output generated by Victor's WIP script, it looks like these are already handled as nicely as those that don't fall under any exception.
Quoted 6 lines
Some (many?) names lack separators between words (example: logappend).
*
UPPER_CASE_WITH_UNDERSCOREfor event names* CamelCase for type names
Quoted 7 lines
Capitalization of words is inconsistent in places (example: VncInfo vs. DisplayReloadOptionsVNC).
What style conversions will we need for Go? Any other conversions come to mind?
What problems do these conversions have?
Go uses CamelCase for pretty much everything: types, methods, constants...
There's one slight wrinkle, in that the case of the first letter decides whether it's going to be a PublicName or a privateName. We can't do anything about that, but it shouldn't really affect us that much because we'll want all QAPI names to be public.
So the issues preventing us from producing a "perfect" Go API are
1. inconsistent capitalization in type names
-> could be addressed by simply changing the schema, as type
names do not travel on the wire
2. missing dashes in certain command/member names
-> leads to Incorrectcamelcase. Kevin's work is supposed to
address this
3. inability to know which parts of a lower-case-name or UPPER_CASE_NAME are acronyms or are otherwise supposed to be capitalized in a specific way
-> leads to WeirdVncAndDbusCapitalization. There's currently no
way, either implemented or planned, to avoid this
In addition to these I'm also thinking that QKeyCode and all the QCrypto stuff should probably lose their prefixes.
Note that 3 shouldn't be an issue for Rust and addressing 1 would actually make things worse for that language, because at the moment at least *some* of the types follow its expected naming rules :)
Quoted 11 lines
Quoted 8 lines
Revised proposal for the annotation:
ns:word-WORD-WoRD-123Word
Words are always separated by dashes; "regular" words are entirely lowercase, while the presence of even a single uppercase letter in a word denotes the fact that its case should be preserved when the naming conventions of the target language allow that.
Is a word always capitalized the same for a single target language? Or could capitalization depend on context?
I'm not aware of any language that would adopt more than a single style of capitalization, outside of course the obvious lower_case_name or UPPER_CASE_NAME scenarios where the original capitalization stops being relevant.
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Am 03.05.2022 um 11:40 hat Andrea Bolognani geschrieben:
Quoted 11 lines
So the issues preventing us from producing a "perfect" Go API are
1. inconsistent capitalization in type names
-> could be addressed by simply changing the schema, as type
names do not travel on the wire2. missing dashes in certain command/member names
-> leads to Incorrectcamelcase. Kevin's work is supposed to
address this
I am surprised that Markus pointed you to my aliases work because the conclusion after reviewing my last attempt was that he doesn't want the feature in the proposed form and he wasn't sure if he wants it at all. To be honest, I considered this one dead.
Even if magically a solution appeared that everyone agreed to, don't hold your breath, I'm working on different things now and not planning to get back to QAPI stuff before 7.1 (and preferably not before meeting Markus in person and discussing the design before I spend the time to produce another thing that will never be merged). I am hoping to get back to QAPI later, but I was mostly planning with continuing the QOM integration, not aliases.
Kevin
Quoted 100 lines · Andrea Bolognani
Quoted 6 lines · Markus Armbruster
Quoted 2 lines · Andrea Bolognani
I still feel that 1) users of a language SDK will ideally not need to look at the QAPI schema or wire chatter too often
I think the most likely point of contact is the QEMU QMP Reference Manual.
Note that there isn't anything preventing us from including the original QAPI name in the documentation for the corresponding Go symbol, or even a link to the reference manual.
So we could make jumping from the Go API documentation, which is what a Go programmer will be looking at most of the time, to the QMP documentation pretty much effortless.
Quoted 4 lines
My point is: a name override feature like the one you propose needs to be used with discipline and restraint. Adds to reviewers' mental load. Needs to be worth it. I'm not saying it isn't, I'm just pointing out a cost.
Yeah, I get that.
Note that I'm not suggesting it should be possible for a name to be completely overridden - I just want to make it possible for a human to provide the name parsing algorithm solutions to those problems it can't figure out on its own.
We could prevent that feature from being misused by verifying that the symbol the annotation is attached to can be derived from the list of words provided. That way, something like
# SOMEName (completely-DIFFERENT-name)
would be rejected and we would avoid misuse.
Quoted 1 line
Wild idea: assume all lower case, but keep a list of exceptions.
That could actually work reasonably well for QEMU because we only need to handle correctly what's in the schema, not arbitrary input.
There's always the risk of the list of exceptions getting out of sync with the needs of the schema, but there's similarly no guarantee that annotations are going to be introduced when they are necessary, so it's mostly a wash.
The only slight advantage of the annotation approach would be that it might be easier to notice it being missing because it's close to the name it refers to, while the list of exceptions is tucked away in a script far away from it.
Quoted 6 lines
The QAPI schema language uses three naming styles:
* lower-case-with-hyphens for command and member names
Many names use upper case and '_'. See pragma command-name-exceptions and member-name-exceptions.
Looking at the output generated by Victor's WIP script, it looks like these are already handled as nicely as those that don't fall under any exception.
Quoted 6 lines
Some (many?) names lack separators between words (example: logappend).
*
UPPER_CASE_WITH_UNDERSCOREfor event names* CamelCase for type names
Quoted 7 lines
Capitalization of words is inconsistent in places (example: VncInfo vs. DisplayReloadOptionsVNC).
What style conversions will we need for Go? Any other conversions come to mind?
What problems do these conversions have?
Go uses CamelCase for pretty much everything: types, methods, constants...
There's one slight wrinkle, in that the case of the first letter decides whether it's going to be a PublicName or a privateName. We can't do anything about that, but it shouldn't really affect us that much because we'll want all QAPI names to be public.
So the issues preventing us from producing a "perfect" Go API are
1. inconsistent capitalization in type names
-> could be addressed by simply changing the schema, as type
names do not travel on the wire2. missing dashes in certain command/member names
-> leads to Incorrectcamelcase. Kevin's work is supposed to
address this3. inability to know which parts of a lower-case-name or
UPPER_CASE_NAMEare acronyms or are otherwise supposed to be capitalized in a specific way
Quoted 5 lines
-> leads to WeirdVncAndDbusCapitalization. There's currently no
way, either implemented or planned, to avoid thisIn addition to these I'm also thinking that QKeyCode and all the QCrypto stuff should probably lose their prefixes.
At the C level, those prefixes are pretty important to avoid clashing with symbols defined by system headers and/or the external crypto library headers, as the unprefixed names are too generic. All non-C languages of course have proper namespacing
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Revised proposal for the annotation:
ns:word-WORD-WoRD-123Word
Words are always separated by dashes; "regular" words are entirely lowercase, while the presence of even a single uppercase letter in a word denotes the fact that its case should be preserved when the naming conventions of the target language allow that.
Quoted 7 lines
Quoted 2 lines
Is a word always capitalized the same for a single target language? Or could capitalization depend on context?
I'm not aware of any language that would adopt more than a single style of capitalization, outside of course the obvious
lower_case_nameorUPPER_CASE_NAMEscenarios where the original capitalization stops being relevant.
As long as the capitalization we use in the schema can provides sufficient info to detect word splitting and acronyms, we can cope with any output language naming rules that I know of.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 28 lines · Andrea Bolognani
Quoted 6 lines · Markus Armbruster
Quoted 2 lines · Andrea Bolognani
I still feel that 1) users of a language SDK will ideally not need to look at the QAPI schema or wire chatter too often
I think the most likely point of contact is the QEMU QMP Reference Manual.
Note that there isn't anything preventing us from including the original QAPI name in the documentation for the corresponding Go symbol, or even a link to the reference manual.
So we could make jumping from the Go API documentation, which is what a Go programmer will be looking at most of the time, to the QMP documentation pretty much effortless.
Quoted 4 lines
My point is: a name override feature like the one you propose needs to be used with discipline and restraint. Adds to reviewers' mental load. Needs to be worth it. I'm not saying it isn't, I'm just pointing out a cost.
Yeah, I get that.
Note that I'm not suggesting it should be possible for a name to be completely overridden - I just want to make it possible for a human to provide the name parsing algorithm solutions to those problems it can't figure out on its own.
Quoted 7 lines
We could prevent that feature from being misused by verifying that the symbol the annotation is attached to can be derived from the list of words provided. That way, something like
# SOMEName (completely-DIFFERENT-name)
would be rejected and we would avoid misuse.
Possibly as simple as "down-case both names and drop the funny characters, result must be the same".
Quoted 10 lines
Quoted 1 line
Wild idea: assume all lower case, but keep a list of exceptions.
That could actually work reasonably well for QEMU because we only need to handle correctly what's in the schema, not arbitrary input.
There's always the risk of the list of exceptions getting out of sync with the needs of the schema, but there's similarly no guarantee that annotations are going to be introduced when they are necessary, so it's mostly a wash.
Quoted 4 lines
The only slight advantage of the annotation approach would be that it might be easier to notice it being missing because it's close to the name it refers to, while the list of exceptions is tucked away in a script far away from it.
We'd put it in qapi/pragma.json, I guess.
Quoted 12 lines
Quoted 6 lines
The QAPI schema language uses three naming styles:
* lower-case-with-hyphens for command and member names
Many names use upper case and '_'. See pragma command-name-exceptions and member-name-exceptions.
Looking at the output generated by Victor's WIP script, it looks like these are already handled as nicely as those that don't fall under any exception.
Quoted 1 line
Some (many?) names lack separators between words (example: logappend).
How many would be good to know.
Ad hoc hackery to find names, filter out camels (because word splitting is too hard there), split into words, look up words in a word list:
$ for i in `/usr/bin/python3 /work/armbru/qemu/scripts/qapi-gen.py -o qapi -b ../qapi/qapi-schema.json | sort -u | awk '/^### [a-z0-9-]+$/ { print "lc", $2; next } /^### [a-z0-9_-]+$/ { print lu; next } /^### [A-Z0-9_]+$/ { print "uc", $2; next } /^### ([A-Z][a-z]+)+/ { print "cc", $2; next } { print "mc", $2 }' | sed '/^mc\|^cc/d;s/^.. //;s/[^A-Za-z0-9]/\n/g' | tr A-Z a-z | sort -u`; do grep -q "^$i$" /usr/share/dict/words || echo $i; done420 lines. How many arguably lack separators between words? Wild guess based on glancing at the output sideways: some 50.
Quoted 20 lines
Quoted 11 lines
*
UPPER_CASE_WITH_UNDERSCOREfor event names* CamelCase for type names
Capitalization of words is inconsistent in places (example: VncInfo vs. DisplayReloadOptionsVNC).
What style conversions will we need for Go? Any other conversions come to mind?
What problems do these conversions have?
Go uses CamelCase for pretty much everything: types, methods, constants...
There's one slight wrinkle, in that the case of the first letter decides whether it's going to be a PublicName or a privateName. We can't do anything about that, but it shouldn't really affect us that much because we'll want all QAPI names to be public.
Quoted 6 lines
So the issues preventing us from producing a "perfect" Go API are
1. inconsistent capitalization in type names
-> could be addressed by simply changing the schema, as type
names do not travel on the wire
At the price of some churn in C code.
Perhaps more consistent capitalization could be regarded as a slight improvement on its own. We need to see (a good sample of) the changes to judge.
Quoted 3 lines
2. missing dashes in certain command/member names
-> leads to Incorrectcamelcase.
Names with words run together are arguably no uglier in CamelCase (Go) than in lower_case_with_underscores (C).
Quoted 2 lines
Kevin's work is supposed to
address this
Except it's stuck.
Perhaps Kevin and I can get it moving again.
Perhaps we can try to extract a local alias feature that can be grown into the more ambitious aliases Kevin wants (if we can solve the issues).
Quoted 6 lines
3. inability to know which parts of a lower-case-name or
UPPER_CASE_NAMEare acronyms or are otherwise supposed to be capitalized in a specific way-> leads to WeirdVncAndDbusCapitalization. There's currently no
way, either implemented or planned, to avoid this
A list of words with special capitalization needs[*]?
VNC is an acronym, some languagues want VNC in camels, some Vnc.
DBus is an abbreviation, some languages want DBus in camels, some Dbus.
Quoted 2 lines
In addition to these I'm also thinking that QKeyCode and all the QCrypto stuff should probably lose their prefixes.
As Daniel pointed out, schema names sometimes have prefixes because we need the generated C identifiers to have prefixes.
If we hate these prefixes enough, we can try to limit them to C identifiers.
Quoted 3 lines
Note that 3 shouldn't be an issue for Rust and addressing 1 would actually make things worse for that language, because at the moment at least *some* of the types follow its expected naming rules :)
Solving Go problems by creating Rust problems doesn't feel like a good move to me.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Revised proposal for the annotation:
ns:word-WORD-WoRD-123Word
Words are always separated by dashes; "regular" words are entirely lowercase, while the presence of even a single uppercase letter in a word denotes the fact that its case should be preserved when the naming conventions of the target language allow that.
Quoted 7 lines
Quoted 2 lines
Is a word always capitalized the same for a single target language? Or could capitalization depend on context?
I'm not aware of any language that would adopt more than a single style of capitalization, outside of course the obvious
lower_case_nameorUPPER_CASE_NAMEscenarios where the original capitalization stops being relevant.
Makes sense.
[*] Sounds like crony capitalism, doesn't it :)
Hi,
Quoted 13 lines · Andrea Bolognani
Quoted 12 lines · Markus Armbruster
Quoted 10 lines · Andrea Bolognani
Quoted 6 lines
Quoted 3 lines
The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.
At the price of mild inconsistency between the library interface and QMP.
That's fine, and in fact it already happens all the time when QAPI names (log-append) are translated to C identifiers (
log_append).
Quoted 8 lines
Quoted 3 lines
There's a difference between trivial translations like "replace '-' by '_'" and arbitrary replacement like the one for enumeration constants involving 'prefix'.
Fair enough.
I still feel that 1) users of a language SDK will ideally not need to look at the QAPI schema or wire chatter too often and
That should be the preference, yes.
Quoted 4 lines
2) even when that ends up being necessary, figuring out that LogAppend and logappend are the same thing is not going to be an unreasonable hurdle, especially when the status quo would be to work with Logappend instead.
If user really needs to leave their ecosystem in order to check an alias, I hope we are already considering this a corner case. Still, if we are thinking about multiple languages communicating with QEMU, it is reasonable to consider the necessity of some docs page where they can easily grep/search for the types, alias, examples, etc. IMHO, this should be a long term goal and not a blocker... I can volunteer on working on that later.
Quoted 52 lines
Quoted 11 lines
Quoted 5 lines
The point is that, if we want to provide a language interface that feels natural, we need a way to mark parts of a QAPI symbol's name in a way that makes it possible for the generator to know they're acronyms and treat them in an appropriate, language-specific manner.
It's not just acronyms. Consider IAmALittleTeapot. If you can assume that only beginning of words are capitalized, even for acronyms, you can split this into words without trouble. You can't recover correct case, though: "i am a little teapot" is wrong.
Is there any scenario in which we would care though? We're in the business of translating identifiers from one machine representation to another, so once it has been split up correctly into the words that compose it (which in your example above it has) then we don't really care about anything else unless acronyms are involved.
In other words, we can obtain the list of words "i am a little teapot" programmatically both from IAmALittleTeapot and i-am-a-little-teapot, and in both cases we can then generate IAmALittleTeapot or
I_AM_A_LITTLE_TEAPOTori_am_a_little_teapotor whatever is appropriate for the context and target language, but the fact that in a proper English sentence "I" would have to be capitalized doesn't really enter the picture.Quoted 4 lines
"Split before capital letter" falls apart when you have characters that cannot be capitalized: Point3d.
Camel case is hopeless.
I would argue that it works quite well for most scenarios, but there are some corner cases where it's clearly not good enough. If we can define a way to clue in the generator about "Point3d" having to be interpreted as "point 3d" and "VNCProps" as "vnc props", then we are golden. That wouldn't be necessary for simple cases that are already handled correctly.
A more radical idea would be to start using dash-notation for types too. That'd remove the word splitting issue altogether, at the cost of the schema being (possibly) harder to read and more distanced from the generated code.
You'd still only be able to generate VncProps from vnc-props though.
Quoted 4 lines
Quoted 3 lines
The obvious way to implement this would be with an annotation along the lines of the one I proposed. Other ideas?
Quoted 6 lines
Quoted 3 lines
I'm afraid having the schema spell out names in multiple naming conventions could be onerous. How many names will need it?
I don't have hard data on this. I could try extracting it, but that might end up being a bigger job than I had anticipated.
The only way to know is by checking /o\
I'll give it a shot.
Quoted 18 lines
My guess is that the number of cases where the naive algorithm can't split words correctly is relatively small compared to the size of the entire QAPI schema. Fair warning: I have made incorrect guesses in the past ;)
Quoted 1 line
Times how many naming conventions?
Yeah, I don't think requiring all possible permutations to be spelled out in the schema is the way to go. That's exactly why my proposal was to offer a way to inject the semantic information that the parser can't figure out itself.
Once you have a way to inform the generator that "VNCProps" is made of the two words "vnc" and "props", and that "vnc" is an acronym, then it can generate an identifier appropriate for the target language without having to spell out anywhere that such an identifier would be VNCProps for Go and VncProps for Rust.
Quoted 8 lines
By the way, while looking around I realized that we also have to take into account things like D-Bus: the QAPI type ChardevDBus, for example, would probably translate verbatim to Go but have to be changed to ChardevDbus for Rust. Fun :)
Revised proposal for the annotation:
ns:word-WORD-WoRD-123Word
I really like it.
Quoted 11 lines
Words are always separated by dashes; "regular" words are entirely lowercase, while the presence of even a single uppercase letter in a word denotes the fact that its case should be preserved when the naming conventions of the target language allow that.
Quoted 3 lines
Another issue: the fancier the translation from schema name to language-specific name gets, the harder it becomes to find one from the other.
That's true, but at least to me the trade-off feels reasonable.
I don't quite get the argument why it gets harder to find. We can simply provide the actual name as reference in the generated documentation, no?
As Kevin already pointed out that he is not planning to work on the Alias (reference below), any other idea besides the Andrea's annotation suggestion? While this is not a blocker, I agree it would be nice to be consistent.
https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg04703.html
Cheers,
Victor
Quoted 96 lines · Victor Toso
Hi,
Quoted 13 lines · Andrea Bolognani
Quoted 12 lines · Markus Armbruster
Quoted 10 lines · Andrea Bolognani
Quoted 6 lines
Quoted 3 lines
The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.
At the price of mild inconsistency between the library interface and QMP.
That's fine, and in fact it already happens all the time when QAPI names (log-append) are translated to C identifiers (
log_append).Quoted 8 lines
Quoted 3 lines
There's a difference between trivial translations like "replace '-' by '_'" and arbitrary replacement like the one for enumeration constants involving 'prefix'.
Fair enough.
I still feel that 1) users of a language SDK will ideally not need to look at the QAPI schema or wire chatter too often and
That should be the preference, yes.
Quoted 4 lines
2) even when that ends up being necessary, figuring out that LogAppend and logappend are the same thing is not going to be an unreasonable hurdle, especially when the status quo would be to work with Logappend instead.
If user really needs to leave their ecosystem in order to check an alias, I hope we are already considering this a corner case. Still, if we are thinking about multiple languages communicating with QEMU, it is reasonable to consider the necessity of some docs page where they can easily grep/search for the types, alias, examples, etc. IMHO, this should be a long term goal and not a blocker... I can volunteer on working on that later.
Quoted 56 lines
Quoted 11 lines
Quoted 5 lines
The point is that, if we want to provide a language interface that feels natural, we need a way to mark parts of a QAPI symbol's name in a way that makes it possible for the generator to know they're acronyms and treat them in an appropriate, language-specific manner.
It's not just acronyms. Consider IAmALittleTeapot. If you can assume that only beginning of words are capitalized, even for acronyms, you can split this into words without trouble. You can't recover correct case, though: "i am a little teapot" is wrong.
Is there any scenario in which we would care though? We're in the business of translating identifiers from one machine representation to another, so once it has been split up correctly into the words that compose it (which in your example above it has) then we don't really care about anything else unless acronyms are involved.
In other words, we can obtain the list of words "i am a little teapot" programmatically both from IAmALittleTeapot and i-am-a-little-teapot, and in both cases we can then generate IAmALittleTeapot or
I_AM_A_LITTLE_TEAPOTori_am_a_little_teapotor whatever is appropriate for the context and target language, but the fact that in a proper English sentence "I" would have to be capitalized doesn't really enter the picture.Quoted 4 lines
"Split before capital letter" falls apart when you have characters that cannot be capitalized: Point3d.
Camel case is hopeless.
I would argue that it works quite well for most scenarios, but there are some corner cases where it's clearly not good enough. If we can define a way to clue in the generator about "Point3d" having to be interpreted as "point 3d" and "VNCProps" as "vnc props", then we are golden. That wouldn't be necessary for simple cases that are already handled correctly.
A more radical idea would be to start using dash-notation for types too. That'd remove the word splitting issue altogether, at the cost of the schema being (possibly) harder to read and more distanced from the generated code.
You'd still only be able to generate VncProps from vnc-props though.
Quoted 7 lines
Quoted 3 lines
The obvious way to implement this would be with an annotation along the lines of the one I proposed. Other ideas?
I'm afraid having the schema spell out names in multiple naming conventions could be onerous. How many names will need it?
Quoted 5 lines
Quoted 2 lines
I don't have hard data on this. I could try extracting it, but that might end up being a bigger job than I had anticipated.
The only way to know is by checking /o\
I'll give it a shot.
I append a quick hack to find names in a QAPI schema, and sort them into buckets "lc" (lower-with-hyphen), "uc" (UPPER_WITH_UNDERSCORE), "cc" (CamelCase where words are obvious), and "mc" (everything else, mostly CamelCase where words aren't obvious).
Note that this ignores naming rule violations such as upper case enum member EAX.
Quoted 39 lines
Quoted 18 lines
My guess is that the number of cases where the naive algorithm can't split words correctly is relatively small compared to the size of the entire QAPI schema. Fair warning: I have made incorrect guesses in the past ;)
Quoted 1 line
Times how many naming conventions?
Yeah, I don't think requiring all possible permutations to be spelled out in the schema is the way to go. That's exactly why my proposal was to offer a way to inject the semantic information that the parser can't figure out itself.
Once you have a way to inform the generator that "VNCProps" is made of the two words "vnc" and "props", and that "vnc" is an acronym, then it can generate an identifier appropriate for the target language without having to spell out anywhere that such an identifier would be VNCProps for Go and VncProps for Rust.
Quoted 8 lines
By the way, while looking around I realized that we also have to take into account things like D-Bus: the QAPI type ChardevDBus, for example, would probably translate verbatim to Go but have to be changed to ChardevDbus for Rust. Fun :)
Revised proposal for the annotation:
ns:word-WORD-WoRD-123Word
I really like it.
Quoted 10 lines
Words are always separated by dashes; "regular" words are entirely lowercase, while the presence of even a single uppercase letter in a word denotes the fact that its case should be preserved when the naming conventions of the target language allow that.
Quoted 3 lines
Another issue: the fancier the translation from schema name to language-specific name gets, the harder it becomes to find one from the other.
Quoted 5 lines
Quoted 1 line
That's true, but at least to me the trade-off feels reasonable.
I don't quite get the argument why it gets harder to find. We can simply provide the actual name as reference in the generated documentation, no?
Predictable name transformation can save me detours through documentation. Being able to guess the Go bindings just from the QMP Reference Manual can be nice: it lets me write Go code with just the QMP Reference manual in view. Being able to guess the name in the QAPI schema from the name in the Go bindings can also be nice: it lets me look it up in the QMP Reference manual without jumping through the bindings documentation.
Or do you envisage a Go bindings manual that fully replaces the QMP Reference Manual for developers writing Go?
Quoted 6 lines
As Kevin already pointed out that he is not planning to work on the Alias (reference below), any other idea besides the Andrea's annotation suggestion? While this is not a blocker, I agree it would be nice to be consistent.
https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg04703.html
Let's first try to get a handle on how many schema names are problematic.
$ git-diff
scripts/qapi/expr.py +1 -0
@@ -94,6 +94,7 @@ def check_name_str(name: str, info: QAPISourceInfo, source: str) -> str: """ # Reserve the entire 'q_' namespace for c_name(), and for 'q_empty' # and 'q_obj_*' implicit type names. + print("###", name) match = valid_name.match(name) if not match or c_name(name, False).startswith('q_'): raise QAPISemError(info, "%s has an invalid name" % source)
$ cd bld
$ python3 ../scripts/qapi-gen.py -o qapi -b ../qapi/qapi-schema.json | sort -u | awk '/^### [a-z0-9-]+$/ { print "lc", $2; next } /^### [A-Z0-9_]+$/ { print "uc", $2; next } /^### ([A-Z][a-z]+)+/ { print "cc", $2; next } { print "mc", $2 }' | sort
cc Abort
cc AbortWrapper
cc AcpiTableOptions
cc ActionCompletionMode
cc AddfdInfo
cc AnnounceParameters
cc AudioFormat
cc Audiodev
cc AudiodevAlsaOptions
cc AudiodevAlsaPerDirectionOptions
cc AudiodevCoreaudioOptions
cc AudiodevCoreaudioPerDirectionOptions
cc AudiodevDriver
cc AudiodevDsoundOptions
cc AudiodevGenericOptions
cc AudiodevJackOptions
cc AudiodevJackPerDirectionOptions
cc AudiodevOssOptions
cc AudiodevOssPerDirectionOptions
cc AudiodevPaOptions
cc AudiodevPaPerDirectionOptions
cc AudiodevPerDirectionOptions
cc AudiodevSdlOptions
cc AudiodevSdlPerDirectionOptions
cc AudiodevWavOptions
cc AuthZListFileProperties
cc AuthZListProperties
cc AuthZPAMProperties
cc AuthZSimpleProperties
cc BackupCommon
cc BackupPerf
cc BalloonInfo
cc BiosAtaTranslation
cc BitmapMigrationBitmapAlias
cc BitmapMigrationBitmapAliasTransform
cc BitmapMigrationNodeAlias
cc BitmapSyncMode
cc BlkdebugEvent
cc BlkdebugIOType
cc BlkdebugInjectErrorOptions
cc BlkdebugSetStateOptions
cc BlockDeviceInfo
cc BlockDeviceIoStatus
cc BlockDeviceStats
cc BlockDeviceTimedStats
cc BlockDirtyBitmap
cc BlockDirtyBitmapAdd
cc BlockDirtyBitmapAddWrapper
cc BlockDirtyBitmapMerge
cc BlockDirtyBitmapMergeWrapper
cc BlockDirtyBitmapOrStr
cc BlockDirtyBitmapSha256
cc BlockDirtyBitmapWrapper
cc BlockDirtyInfo
cc BlockErrorAction
cc BlockExportInfo
cc BlockExportOptions
cc BlockExportOptionsFuse
cc BlockExportOptionsNbd
cc BlockExportOptionsNbdBase
cc BlockExportOptionsVhostUserBlk
cc BlockExportRemoveMode
cc BlockExportType
cc BlockIOThrottle
cc BlockInfo
cc BlockJobInfo
cc BlockLatencyHistogramInfo
cc BlockMeasureInfo
cc BlockPermission
cc BlockStats
cc BlockStatsSpecific
cc BlockStatsSpecificFile
cc BlockStatsSpecificNvme
cc BlockdevAioOptions
cc BlockdevAmendOptions
cc BlockdevAmendOptionsLUKS
cc BlockdevAmendOptionsQcow2
cc BlockdevBackup
cc BlockdevBackupWrapper
cc BlockdevCacheInfo
cc BlockdevCacheOptions
cc BlockdevChangeReadOnlyMode
cc BlockdevCreateOptions
cc BlockdevCreateOptionsFile
cc BlockdevCreateOptionsGluster
cc BlockdevCreateOptionsLUKS
cc BlockdevCreateOptionsNfs
cc BlockdevCreateOptionsParallels
cc BlockdevCreateOptionsQcow
cc BlockdevCreateOptionsQcow2
cc BlockdevCreateOptionsQed
cc BlockdevCreateOptionsRbd
cc BlockdevCreateOptionsSsh
cc BlockdevCreateOptionsVdi
cc BlockdevCreateOptionsVhdx
cc BlockdevCreateOptionsVmdk
cc BlockdevCreateOptionsVpc
cc BlockdevDetectZeroesOptions
cc BlockdevDiscardOptions
cc BlockdevDriver
cc BlockdevOnError
cc BlockdevOptions
cc BlockdevOptionsBlkdebug
cc BlockdevOptionsBlklogwrites
cc BlockdevOptionsBlkreplay
cc BlockdevOptionsBlkverify
cc BlockdevOptionsCbw
cc BlockdevOptionsCor
cc BlockdevOptionsCurlBase
cc BlockdevOptionsCurlFtp
cc BlockdevOptionsCurlFtps
cc BlockdevOptionsCurlHttp
cc BlockdevOptionsCurlHttps
cc BlockdevOptionsFile
cc BlockdevOptionsGenericCOWFormat
cc BlockdevOptionsGenericFormat
cc BlockdevOptionsGluster
cc BlockdevOptionsIscsi
cc BlockdevOptionsLUKS
cc BlockdevOptionsNVMe
cc BlockdevOptionsNbd
cc BlockdevOptionsNfs
cc BlockdevOptionsNull
cc BlockdevOptionsPreallocate
cc BlockdevOptionsQcow
cc BlockdevOptionsQcow2
cc BlockdevOptionsQuorum
cc BlockdevOptionsRaw
cc BlockdevOptionsRbd
cc BlockdevOptionsReplication
cc BlockdevOptionsSsh
cc BlockdevOptionsThrottle
cc BlockdevOptionsVVFAT
cc BlockdevQcow2Encryption
cc BlockdevQcow2EncryptionFormat
cc BlockdevQcow2Version
cc BlockdevQcowEncryption
cc BlockdevQcowEncryptionFormat
cc BlockdevRef
cc BlockdevRefOrNull
cc BlockdevSnapshot
cc BlockdevSnapshotInternal
cc BlockdevSnapshotInternalWrapper
cc BlockdevSnapshotSync
cc BlockdevSnapshotSyncWrapper
cc BlockdevSnapshotWrapper
cc BlockdevVhdxSubformat
cc BlockdevVmdkAdapterType
cc BlockdevVmdkSubformat
cc BlockdevVpcSubformat
cc CanHostSocketcanProperties
cc ChardevBackend
cc ChardevBackendInfo
cc ChardevBackendKind
cc ChardevCommon
cc ChardevCommonWrapper
cc ChardevDBus
cc ChardevDBusWrapper
cc ChardevFile
cc ChardevFileWrapper
cc ChardevHostdev
cc ChardevHostdevWrapper
cc ChardevInfo
cc ChardevMux
cc ChardevMuxWrapper
cc ChardevQemuVDAgent
cc ChardevQemuVDAgentWrapper
cc ChardevReturn
cc ChardevRingbuf
cc ChardevRingbufWrapper
cc ChardevSocket
cc ChardevSocketWrapper
cc ChardevSpiceChannel
cc ChardevSpiceChannelWrapper
cc ChardevSpicePort
cc ChardevSpicePortWrapper
cc ChardevStdio
cc ChardevStdioWrapper
cc ChardevUdp
cc ChardevUdpWrapper
cc ChardevVC
cc ChardevVCWrapper
cc ColoCompareProperties
cc CommandInfo
cc CommandLineOptionInfo
cc CommandLineParameterInfo
cc CommandLineParameterType
cc CommandNotFound
cc CompatPolicy
cc CompatPolicyInput
cc CompatPolicyOutput
cc CompressionStats
cc CpuDefinitionInfo
cc CpuInfoFast
cc CpuInfoS390
cc CpuInstanceProperties
cc CpuModelBaselineInfo
cc CpuModelCompareInfo
cc CpuModelCompareResult
cc CpuModelExpansionInfo
cc CpuModelExpansionType
cc CpuModelInfo
cc CpuS390State
cc CryptodevBackendProperties
cc CryptodevVhostUserProperties
cc CurrentMachineParams
cc DataFormat
cc DeviceNotActive
cc DeviceNotFound
cc DirtyRateInfo
cc DirtyRateMeasureMode
cc DirtyRateStatus
cc DirtyRateVcpu
cc DisplayCocoa
cc DisplayCurses
cc DisplayDBus
cc DisplayEGLHeadless
cc DisplayGLMode
cc DisplayGTK
cc DisplayOptions
cc DisplayProtocol
cc DisplayReloadOptions
cc DisplayReloadOptionsVNC
cc DisplayReloadType
cc DisplayType
cc DisplayUpdateOptions
cc DisplayUpdateOptionsVNC
cc DisplayUpdateType
cc DriveBackup
cc DriveBackupWrapper
cc DriveMirror
cc DummyForceArrays
cc DumpGuestMemoryCapability
cc DumpGuestMemoryFormat
cc DumpQueryResult
cc DumpStatus
cc EventLoopBaseProperties
cc ExpirePasswordOptions
cc ExpirePasswordOptionsVnc
cc FailoverStatus
cc FdsetFdInfo
cc FdsetInfo
cc FilterBufferProperties
cc FilterDumpProperties
cc FilterMirrorProperties
cc FilterRedirectorProperties
cc FilterRewriterProperties
cc FloppyDriveType
cc FuseExportAllowOther
cc GenericError
cc GrabToggleKeys
cc GuestPanicAction
cc GuestPanicInformation
cc GuestPanicInformationHyperV
cc GuestPanicInformationS390
cc GuestPanicInformationType
cc GuidInfo
cc HmatCacheAssociativity
cc HmatCacheWritePolicy
cc HmatLBDataType
cc HmatLBMemoryHierarchy
cc HostMemPolicy
cc HotpluggableCPU
cc HumanReadableText
cc ImageCheck
cc ImageFormat
cc ImageInfo
cc ImageInfoSpecific
cc ImageInfoSpecificKind
cc ImageInfoSpecificLUKSWrapper
cc ImageInfoSpecificQCow2
cc ImageInfoSpecificQCow2Encryption
cc ImageInfoSpecificQCow2EncryptionBase
cc ImageInfoSpecificQCow2Wrapper
cc ImageInfoSpecificRbd
cc ImageInfoSpecificRbdWrapper
cc ImageInfoSpecificVmdk
cc ImageInfoSpecificVmdkWrapper
cc InetSocketAddress
cc InetSocketAddressBase
cc InetSocketAddressWrapper
cc InputAxis
cc InputBarrierProperties
cc InputBtnEvent
cc InputBtnEventWrapper
cc InputButton
cc InputEvent
cc InputEventKind
cc InputKeyEvent
cc InputKeyEventWrapper
cc InputLinuxProperties
cc InputMoveEvent
cc InputMoveEventWrapper
cc IntWrapper
cc IoOperationType
cc IothreadProperties
cc IscsiHeaderDigest
cc IscsiTransport
cc JobInfo
cc JobStatus
cc JobType
cc JobVerb
cc KeyValue
cc KeyValueKind
cc KvmInfo
cc LostTickPolicy
cc MachineInfo
cc MainLoopProperties
cc MapEntry
cc Memdev
cc MemoryBackendEpcProperties
cc MemoryBackendFileProperties
cc MemoryBackendMemfdProperties
cc MemoryBackendProperties
cc MemoryDeviceInfo
cc MemoryDeviceInfoKind
cc MemoryFailureAction
cc MemoryFailureFlags
cc MemoryFailureRecipient
cc MemoryInfo
cc MigrateSetParameters
cc MigrationCapability
cc MigrationCapabilityStatus
cc MigrationInfo
cc MigrationParameter
cc MigrationParameters
cc MigrationStats
cc MigrationStatus
cc MirrorCopyMode
cc MirrorSyncMode
cc MonitorMode
cc MonitorOptions
cc MouseInfo
cc MultiFDCompression
cc NameInfo
cc NbdServerAddOptions
cc NbdServerOptions
cc NetClientDriver
cc NetFilterDirection
cc NetLegacyNicOptions
cc Netdev
cc NetdevBridgeOptions
cc NetdevHubPortOptions
cc NetdevL2TPv3Options
cc NetdevNetmapOptions
cc NetdevSocketOptions
cc NetdevTapOptions
cc NetdevUserOptions
cc NetdevVdeOptions
cc NetdevVhostUserOptions
cc NetdevVhostVDPAOptions
cc NetfilterInsert
cc NetfilterProperties
cc NetworkAddressFamily
cc NewImageMode
cc NumaCpuOptions
cc NumaDistOptions
cc NumaHmatCacheOptions
cc NumaHmatLBOptions
cc NumaNodeOptions
cc NumaOptions
cc NumaOptionsType
cc ObjectOptions
cc ObjectPropertyInfo
cc ObjectType
cc ObjectTypeInfo
cc OffAutoPCIBAR
cc OnOffAuto
cc OnOffSplit
cc PanicAction
cc PciBridgeInfo
cc PciBusInfo
cc PciDeviceClass
cc PciDeviceId
cc PciDeviceInfo
cc PciInfo
cc PciMemoryRange
cc PciMemoryRegion
cc PrManagerHelperProperties
cc PreallocMode
cc QapiErrorClass
cc Qcow2BitmapInfo
cc Qcow2BitmapInfoFlags
cc Qcow2CompressionType
cc Qcow2OverlapCheckFlags
cc Qcow2OverlapCheckMode
cc Qcow2OverlapChecks
cc QtestProperties
cc QuorumOpType
cc QuorumReadPattern
cc RbdAuthMode
cc RbdEncryptionCreateOptions
cc RbdEncryptionCreateOptionsLUKS
cc RbdEncryptionCreateOptionsLUKS2
cc RbdEncryptionCreateOptionsLUKSBase
cc RbdEncryptionOptions
cc RbdEncryptionOptionsLUKS
cc RbdEncryptionOptionsLUKS2
cc RbdEncryptionOptionsLUKSBase
cc RbdImageEncryptionFormat
cc RebootAction
cc RemoteObjectProperties
cc ReplayInfo
cc ReplayMode
cc ReplicationMode
cc ReplicationStatus
cc RngEgdProperties
cc RngProperties
cc RngRandomProperties
cc RockerOfDpaFlow
cc RockerOfDpaFlowAction
cc RockerOfDpaFlowKey
cc RockerOfDpaFlowMask
cc RockerOfDpaGroup
cc RockerPort
cc RockerPortAutoneg
cc RockerPortDuplex
cc RockerSwitch
cc RunState
cc RxFilterInfo
cc RxState
cc SchemaInfo
cc SchemaInfoAlternate
cc SchemaInfoAlternateMember
cc SchemaInfoArray
cc SchemaInfoBuiltin
cc SchemaInfoCommand
cc SchemaInfoEnum
cc SchemaInfoEnumMember
cc SchemaInfoEvent
cc SchemaInfoObject
cc SchemaInfoObjectMember
cc SchemaInfoObjectVariant
cc SchemaMetaType
cc SecretCommonProperties
cc SecretKeyringProperties
cc SecretProperties
cc SetPasswordAction
cc SetPasswordOptions
cc SetPasswordOptionsVnc
cc SevAttestationReport
cc SevCapability
cc SevGuestProperties
cc SevInfo
cc SevLaunchMeasureInfo
cc SevState
cc SgxEPC
cc SgxEPCDeviceInfo
cc SgxEPCDeviceInfoWrapper
cc SgxEPCProperties
cc ShutdownAction
cc ShutdownCause
cc SmbiosEntryPointType
cc SnapshotInfo
cc SocketAddress
cc SocketAddressLegacy
cc SocketAddressType
cc SpiceBasicInfo
cc SpiceChannel
cc SpiceInfo
cc SpiceQueryMouseMode
cc SpiceServerInfo
cc SshHostKeyCheck
cc SshHostKeyCheckHashType
cc SshHostKeyCheckMode
cc SshHostKeyHash
cc StatusInfo
cc StrOrNull
cc String
cc StringWrapper
cc SysEmuTarget
cc TargetInfo
cc ThrottleGroupProperties
cc ThrottleLimits
cc TlsCredsAnonProperties
cc TlsCredsProperties
cc TlsCredsPskProperties
cc TlsCredsX509Properties
cc TpmModel
cc TpmType
cc TpmTypeOptions
cc TraceEventInfo
cc TraceEventState
cc TransactionAction
cc TransactionActionKind
cc TransactionProperties
cc UnixSocketAddress
cc UnixSocketAddressWrapper
cc UuidInfo
cc VersionInfo
cc VersionTriple
cc VfioStats
cc VirtioMEMDeviceInfo
cc VirtioMEMDeviceInfoWrapper
cc VirtioPMEMDeviceInfo
cc VirtioPMEMDeviceInfoWrapper
cc VncBasicInfo
cc VncClientInfo
cc VncInfo
cc VncInfo2
cc VncPrimaryAuth
cc VncServerInfo
cc VncServerInfo2
cc VncVencryptSubAuth
cc VsockSocketAddress
cc VsockSocketAddressWrapper
cc WatchdogAction
cc YankInstance
cc YankInstanceBlockNode
cc YankInstanceChardev
cc YankInstanceType
lc a
lc aarch64
lc abort
lc aborting
lc abs
lc absolute
lc absolute-paths
lc abstract
lc accept
lc access-bandwidth
lc access-latency
lc action
lc action-required
lc actions
lc active
lc active-l1
lc active-l2
lc actual
lc actual-size
lc adapter-type
lc add-fd
lc addr
lc address
lc addresses
lc aes
lc aes-128
lc aes-192
lc aes-256
lc again
lc aio
lc aio-max-batch
lc alias
lc alias-of
lc align
lc aligned-accesses
lc all
lc allocated-clusters
lc allocation-depth
lc allow
lc allow-oob
lc allow-other
lc allow-write-only-overlay
lc alpha
lc alsa
lc alt
lc alt-alt
lc alternate
lc always
lc amend
lc amount-exceeded
lc announce-initial
lc announce-max
lc announce-rounds
lc announce-self
lc announce-step
lc api-major
lc api-minor
lc apostrophe
lc append
lc arch
lc arg-type
lc arg1
lc arg2
lc arg3
lc arg4
lc arg5
lc arm
lc array
lc associativity
lc asterisk
lc audiodev
lc audiomute
lc audionext
lc audioplay
lc audioprev
lc audiostop
lc auth
lc auth-client-required
lc authz-list
lc authz-listfile
lc authz-pam
lc authz-simple
lc auto
lc auto-converge
lc auto-dismiss
lc auto-finalize
lc auto-read-only
lc autoneg
lc avr
lc axis
lc b
lc backend
lc background
lc background-snapshot
lc backing
lc backing-file
lc backing-filename
lc backing-filename-format
lc backing-fmt
lc backing-image
lc backslash
lc backspace
lc backup
lc balloon
lc bandwidth
lc bar
lc bar0
lc bar1
lc bar2
lc bar3
lc bar4
lc bar5
lc base
lc base-memory
lc base-node
lc base64
lc before
lc begin
lc behind
lc bind
lc bins
lc bitmap
lc bitmap-directory
lc bitmap-mode
lc bitmaps
lc blk
lc blkdebug
lc blklogwrites
lc blkreplay
lc blkverify
lc block
lc block-backend
lc block-bitmap-mapping
lc block-commit
lc block-dirty-bitmap-add
lc block-dirty-bitmap-clear
lc block-dirty-bitmap-disable
lc block-dirty-bitmap-enable
lc block-dirty-bitmap-merge
lc block-dirty-bitmap-remove
lc block-driver
lc block-export-add
lc block-export-del
lc block-incremental
lc block-job
lc block-job-cancel
lc block-job-complete
lc block-job-dismiss
lc block-job-finalize
lc block-job-pause
lc block-job-resume
lc block-job-set-speed
lc block-latency-histogram-set
lc block-node
lc block-set-write-threshold
lc block-size
lc block-state-zero
lc block-status
lc block-stream
lc blockdev-add
lc blockdev-backup
lc blockdev-change-medium
lc blockdev-close-tray
lc blockdev-create
lc blockdev-del
lc blockdev-insert-medium
lc blockdev-mirror
lc blockdev-open-tray
lc blockdev-remove-medium
lc blockdev-reopen
lc blockdev-snapshot
lc blockdev-snapshot-delete-internal-sync
lc blockdev-snapshot-internal-sync
lc blockdev-snapshot-sync
lc blocked-reasons
lc bochs
lc boolean
lc bootfile
lc bottom
lc boundaries
lc boundaries-flush
lc boundaries-read
lc boundaries-write
lc bps
lc bps-read
lc bps-read-max
lc bps-read-max-length
lc bps-total
lc bps-total-max
lc bps-total-max-length
lc bps-write
lc bps-write-max
lc bps-write-max-length
lc br
lc braille
lc bridge
lc broadcast-allowed
lc btn
lc buf-size
lc buffer-count
lc buffer-length
lc build-id
lc builtin
lc bus
lc buslogic
lc busy
lc busy-rate
lc button
lc bytes
lc c
lc cache
lc cache-clean-interval
lc cache-miss
lc cache-miss-rate
lc cache-size
lc cached
lc calc-dirty-rate
lc calc-time
lc calculator
lc can-bus
lc can-host-socketcan
lc canbus
lc cancel
lc cancel-path
lc cancelled
lc cancelling
lc capabilities
lc capability
lc case
lc cast5-128
lc cbc
lc cbitpos
lc cephx
lc cert-chain
lc cert-subject
lc change-backing-file
lc change-vnc-password
lc channel-id
lc channel-type
lc channels
lc chardev
lc chardev-add
lc chardev-change
lc chardev-remove
lc chardev-send-break
lc charset
lc check-errors
lc check-stop
lc checkpoint-ready
lc checkpoint-reply
lc checkpoint-request
lc child
lc children
lc cid
lc cipher-alg
lc cipher-mode
lc class
lc client
lc client-name
lc clients
lc clipboard
lc cloop
lc closefd
lc cluster-id
lc cluster-size
lc clusters
lc cocoa
lc colo
lc colo-compare
lc cols
lc comma
lc command
lc command-line
lc commit
lc compat
lc compiled-version
lc complete
lc completed
lc completion-errors
lc completion-mode
lc complex
lc compose
lc compress
lc compress-level
lc compress-threads
lc compress-wait-thread
lc compressed
lc compressed-clusters
lc compressed-size
lc compression
lc compression-rate
lc compression-type
lc computer
lc concluded
lc conf
lc config
lc connect
lc connect-ports
lc connected
lc connection-id
lc consistent-read
lc console
lc constant
lc cont
lc control
lc cookie
lc cookie-secret
lc cookie64
lc copy
lc copy-before-write
lc copy-mode
lc copy-on-read
lc core
lc core-id
lc coreaudio
lc cores
lc corrupt
lc corruptions
lc corruptions-fixed
lc count
lc counter
lc cpu
lc cpu-index
lc cpu-max
lc cpu-state
lc cpu-throttle-increment
lc cpu-throttle-initial
lc cpu-throttle-percentage
lc cpu-throttle-tailslow
lc cpu0-id
lc cpuid-input-eax
lc cpuid-input-ecx
lc cpuid-register
lc cpus
lc crash
lc crc32c
lc crc32c-none
lc create
lc create-type
lc created
lc cris
lc cryptodev-backend
lc cryptodev-backend-builtin
lc cryptodev-vhost-user
lc ctr
lc ctrl
lc ctrl-ctrl
lc ctrl-scrolllock
lc current
lc current-progress
lc curses
lc cut
lc d
lc d0
lc d1
lc d12
lc d120
lc d144
lc d16
lc d2
lc d288
lc d3
lc d32
lc d3des
lc d4
lc d5
lc d6
lc d64
lc d7
lc d8
lc d9
lc data
lc data-file
lc data-file-raw
lc data-type
lc date-nsec
lc date-sec
lc dbus
lc dbus-vmstate
lc debug
lc decompress-threads
lc default
lc default-cpu-type
lc default-ram-id
lc default-value
lc definition
lc delay
lc delete
lc deny
lc deprecated
lc deprecated-input
lc deprecated-output
lc depth
lc des
lc desc
lc description
lc detach
lc detect-zeroes
lc dev
lc device
lc device-id
lc device-list-properties
lc devices
lc devid
lc devname
lc dh-cert-file
lc dhcpstart
lc die-id
lc dies
lc dimm
lc dir
lc direct
lc dirty-bitmap
lc dirty-bitmaps
lc dirty-flag
lc dirty-pages-rate
lc dirty-rate
lc dirty-ring
lc dirty-sync-count
lc disabled
lc disabled-wait
lc discard
lc discard-bytes-ok
lc discard-data
lc discard-nb-failed
lc discard-nb-ok
lc disconnect
lc disk
lc dismiss
lc display
lc display-reload
lc display-update
lc dist
lc dmg
lc dns
lc dnssearch
lc domainname
lc dot
lc down
lc downscript
lc downtime
lc downtime-bytes
lc downtime-limit
lc drive-backup
lc drive-mirror
lc driver
lc driver-specific
lc drop-cache
lc drv
lc dsound
lc dsp-policy
lc dst
lc dstport
lc dump
lc dump-guest-memory
lc dump-skeys
lc duplex
lc duplicate
lc dynamic
lc dynamic-auto-read-only
lc e
lc ecb
lc edges
lc egl-headless
lc eject
lc element-type
lc elf
lc emulated
lc emulator
lc enable
lc enabled
lc encoding-rate
lc encrypt
lc encrypted
lc encryption-format
lc end
lc endpoint
lc enospc
lc enum
lc equal
lc errno
lc error
lc error-desc
lc es
lc esc
lc essiv
lc eth-dst
lc eth-src
lc eth-type
lc evdev
lc event
lc events
lc exact
lc exact-name
lc exclusive
lc existing
lc expected-downtime
lc export
lc extended-l2
lc extent-size-hint
lc extents
lc external
lc extint-loop
lc extra
lc f
lc f1
lc f10
lc f11
lc f12
lc f2
lc f3
lc f32
lc f4
lc f5
lc f6
lc f7
lc f8
lc f9
lc fail
lc failed
lc failover
lc falloc
lc family
lc fat-type
lc fatal
lc fd
lc fdname
lc fds
lc fdset-id
lc features
lc fifo
lc file
lc filename
lc filter-buffer
lc filter-dump
lc filter-mirror
lc filter-node-name
lc filter-redirector
lc filter-replay
lc filter-rewriter
lc finalize
lc find
lc finish-migrate
lc first-level
lc fixed
lc fixed-iothread
lc fixed-settings
lc flags
lc flat
lc flc
lc floppy
lc flush
lc force
lc force-share
lc force-size
lc format
lc format-specific
lc formats
lc fqdn
lc fragmented-clusters
lc frequency
lc front
lc frontend-open
lc ftp
lc ftps
lc full
lc full-backing-filename
lc full-grab
lc full-screen
lc fully-allocated
lc function
lc fuse
lc g
lc getfd
lc gid-status
lc gl
lc glob
lc gluster
lc goto-tbl
lc gpa
lc grab-on-hover
lc grab-toggle
lc granularity
lc group
lc group-id
lc group-ids
lc grouped
lc growable
lc gtk
lc guest
lc guest-panic
lc guest-panicked
lc guest-reset
lc guest-shutdown
lc guestfwd
lc guid
lc h
lc half
lc handle
lc hard
lc hash
lc hash-alg
lc head
lc header-digest
lc height
lc help
lc helper
lc henkan
lc hide
lc hierarchy
lc hiragana
lc hits
lc hmat-cache
lc hmat-lb
lc hold-time
lc home
lc host
lc host-error
lc host-key-check
lc host-nodes
lc host-qmp-quit
lc host-qmp-system-reset
lc host-signal
lc host-ui
lc hostfwd
lc hostname
lc hotpluggable
lc hotpluggable-cpus
lc hotplugged
lc hppa
lc http
lc https
lc hubid
lc hubport
lc hugetlb
lc hugetlbsize
lc human-monitor-command
lc human-readable-text
lc hwversion
lc hyper-v
lc hypervisor
lc i
lc i386
lc icount
lc id
lc id-list
lc ide
lc identical
lc identity
lc if
lc ifname
lc ignore
lc ignore-unavailable
lc image
lc image-end-offset
lc image-node-name
lc immediately
lc implements
lc in
lc in-pport
lc in-use
lc inactive
lc inactive-l1
lc inactive-l2
lc inc
lc incompatible
lc inconsistent
lc incremental
lc indev
lc index
lc individual
lc inet
lc info
lc initial
lc initiator
lc initiator-name
lc inject
lc inject-error
lc inject-nmi
lc inmigrate
lc input-barrier
lc input-linux
lc input-send-event
lc insert
lc inserted
lc instances
lc int
lc interface-id
lc interfaces
lc interleave
lc internal-error
lc interval
lc io-error
lc io-status
lc iops
lc iops-read
lc iops-read-max
lc iops-read-max-length
lc iops-size
lc iops-total
lc iops-total-max
lc iops-total-max-length
lc iops-write
lc iops-write-max
lc iops-write-max-length
lc iothread
lc iotype
lc ip
lc ip-dst
lc ip-proto
lc ip-tos
lc ipv4
lc ipv6
lc ipv6-dns
lc ipv6-host
lc ipv6-prefix
lc ipv6-prefixlen
lc irq
lc is-default
lc iscsi
lc iser
lc iter-time
lc iters
lc iv
lc ivgen-alg
lc ivgen-hash-alg
lc j
lc jack
lc job-cancel
lc job-complete
lc job-dismiss
lc job-finalize
lc job-id
lc job-pause
lc job-resume
lc json-cli
lc json-cli-hotplug
lc json-type
lc k
lc katakanahiragana
lc kdump-lzo
lc kdump-snappy
lc kdump-zlib
lc keep
lc keep-alive
lc kernel
lc kernel-hashes
lc key
lc key-offset
lc key-secret
lc keyid
lc keys
lc keyslot
lc l
lc l2-cache-entry-size
lc l2-cache-size
lc l2tpv3
lc label
lc lang1
lc lang2
lc large
lc last-mode
lc late-block-activate
lc latency
lc latency-ns
lc launch-secret
lc launch-update
lc lazy-refcounts
lc lba
lc leaks
lc leaks-fixed
lc left
lc left-command-key
lc len
lc length
lc less
lc level
lc lf
lc limit
lc limits
lc line
lc link-up
lc listen
lc live
lc load
lc loaded
lc local
lc localaddr
lc location
lc locked
lc locking
lc log
lc log-append
lc log-sector-size
lc log-size
lc log-super-update-interval
lc logappend
lc logfile
lc logical-block-size
lc lsilogic
lc luks
lc luks2
lc lun
lc m
lc m68k
lc macaddr
lc mail
lc main-header
lc main-loop
lc main-mac
lc major
lc mask
lc master-key-iters
lc match
lc max
lc max-bandwidth
lc max-chunk
lc max-connections
lc max-cpu-throttle
lc max-discard
lc max-postcopy-bandwidth
lc max-size
lc max-transfer
lc max-workers
lc max-write-zero
lc maxcpus
lc maxlen
lc mbps
lc mcast
lc md5
lc measured
lc measuring
lc mediaselect
lc mem
lc mem-path
lc memaddr
lc members
lc memdev
lc memory
lc memory-backend-epc
lc memory-backend-file
lc memory-backend-memfd
lc memory-backend-ram
lc memsave
lc menu
lc merge
lc meta-meta
lc meta-type
lc metadata
lc micro
lc microblaze
lc microblazeel
lc middle
lc migrate
lc migrate-continue
lc migrate-incoming
lc migrate-pause
lc migrate-recover
lc migrate-set-capabilities
lc migrate-set-parameters
lc migrate-start-postcopy
lc migrated
lc migration
lc migration-safe
lc minor
lc minus
lc mips
lc mips64
lc mips64el
lc mipsel
lc mirror
lc mixing-engine
lc mnonce
lc mode
lc model
lc modela
lc modelb
lc mountpoint
lc mouse
lc mouse-mode
lc mptcp
lc msg
lc msmouse
lc muhenkan
lc multicast
lc multicast-overflow
lc multicast-table
lc multifd
lc multifd-bytes
lc multifd-channels
lc multifd-compression
lc multifd-zlib-level
lc multifd-zstd-level
lc mux
lc n
lc name
lc namespace
lc native
lc nbd
lc nbd-server-add
lc nbd-server-remove
lc nbd-server-start
lc nbd-server-stop
lc net
lc netdev
lc netmap
lc never
lc new-secret
lc new-vlan-id
lc nfs
lc nic
lc nios2
lc no-flush
lc nocow
lc node
lc node-id
lc node-name
lc nodeid
lc nodelay
lc nodes
lc none
lc none-crc32c
lc normal
lc normal-bytes
lc nospace
lc null
lc null-aio
lc null-co
lc num-queues
lc numa-mem-supported
lc number
lc numeric
lc nvdimm
lc nvme
lc o
lc object
lc object-add
lc object-del
lc off
lc offset
lc ok
lc old-secret
lc on
lc on-error
lc on-source-error
lc on-success
lc on-target-error
lc once
lc oob
lc opaque
lc open
lc open-timeout
lc opened
lc operating
lc operation
lc opint-loop
lc opt-discard
lc opt-write-zero
lc option
lc options
lc or1k
lc oss
lc out
lc out-pport
lc outdev
lc overflow
lc overlap-check
lc overlay
lc p
lc p2p
lc pa
lc package
lc packet-header
lc page-cache-size
lc page-sampling
lc page-size
lc pages
lc pages-per-second
lc paging
lc panic
lc parallel
lc parallels
lc parameters
lc parent
lc parent-cid
lc pass
lc pass-discard-other
lc pass-discard-request
lc pass-discard-snapshot
lc passthrough
lc password
lc password-secret
lc passwordid
lc paste
lc path
lc pause
lc pause-before-switchover
lc paused
lc payload-offset
lc pdh
lc pef-guest
lc pending
lc period-length
lc perm
lc persistent
lc pgdn
lc pgmint-loop
lc pgup
lc pincounter
lc pipe
lc plain
lc plain64
lc play
lc plugged-memory
lc pmem
lc pmemsave
lc png
lc policy
lc poll-grow
lc poll-max-ns
lc poll-shrink
lc poll-us
lc pool
lc pop-vlan
lc port
lc portal
lc ports
lc position
lc postcopy-active
lc postcopy-blocktime
lc postcopy-bytes
lc postcopy-paused
lc postcopy-ram
lc postcopy-recover
lc postcopy-requests
lc postcopy-vcpu-blocktime
lc postmigrate
lc power
lc poweroff
lc ppc
lc ppc64
lc ppm
lc pport
lc pr-manager
lc pr-manager-helper
lc pre-switchover
lc prealloc
lc prealloc-align
lc prealloc-size
lc prealloc-threads
lc preallocate
lc preallocation
lc precopy-bytes
lc preferred
lc prefetch
lc prelaunch
lc present
lc pretty
lc primary
lc print
lc priority
lc processing
lc promiscuous
lc properties
lc property
lc props
lc protocol
lc proxy-password-secret
lc proxy-username
lc psw-addr
lc psw-mask
lc pty
lc pwritev
lc q
lc qcode
lc qcow
lc qcow2
lc qdev
lc qed
lc qemu
lc qemu-vdagent
lc qom-get
lc qom-list
lc qom-list-properties
lc qom-list-types
lc qom-path
lc qom-set
lc qom-type
lc qtest
lc query-acpi-ospm-status
lc query-balloon
lc query-block
lc query-block-exports
lc query-block-jobs
lc query-blockstats
lc query-chardev
lc query-chardev-backends
lc query-colo-status
lc query-command-line-options
lc query-commands
lc query-cpu-definitions
lc query-cpu-model-baseline
lc query-cpu-model-comparison
lc query-cpu-model-expansion
lc query-cpus-fast
lc query-current-machine
lc query-dirty-rate
lc query-display-options
lc query-dump
lc query-dump-guest-memory-capability
lc query-fdsets
lc query-gic-capabilities
lc query-hotpluggable-cpus
lc query-iothreads
lc query-jobs
lc query-kvm
lc query-machines
lc query-memdev
lc query-memory-devices
lc query-memory-size-summary
lc query-mice
lc query-migrate
lc query-migrate-capabilities
lc query-migrate-parameters
lc query-name
lc query-named-block-nodes
lc query-nodes
lc query-pci
lc query-pr-managers
lc query-qmp-schema
lc query-replay
lc query-rocker
lc query-rocker-of-dpa-flows
lc query-rocker-of-dpa-groups
lc query-rocker-ports
lc query-rx-filter
lc query-sev
lc query-sev-attestation-report
lc query-sev-capabilities
lc query-sev-launch-measure
lc query-sgx
lc query-sgx-capabilities
lc query-spice
lc query-status
lc query-target
lc query-tpm
lc query-tpm-models
lc query-tpm-types
lc query-uuid
lc query-version
lc query-vm-generation-id
lc query-vnc
lc query-vnc-servers
lc query-xen-replication-status
lc query-yank
lc queue
lc queues
lc quit
lc quorum
lc r
lc ra2
lc ra2ne
lc ram
lc raw
lc rbd
lc rdma-pin-all
lc read
lc read-bandwidth
lc read-latency
lc read-only
lc read-only-mode
lc read-pattern
lc read-write
lc read-zeroes
lc readahead
lc readahead-size
lc readline
lc readonly
lc ready
lc reason
lc reboot
lc receive-update
lc rechs
lc recipient
lc reconnect
lc reconnect-delay
lc record
lc recording
lc recursive
lc reduced-phys-bits
lc refcount-bits
lc refcount-block
lc refcount-cache-size
lc refcount-table
lc reference
lc refresh
lc regions
lc reject
lc rel
lc relaunch
lc release-ram
lc remaining
lc remote
lc removable
lc remove-fd
lc rendernode
lc repeat
lc replaces
lc replay-break
lc replay-delete-break
lc replay-seek
lc replication
lc report
lc request
lc requested-size
lc require
lc required
lc reserve
lc reset
lc resize
lc responsible-properties
lc restore-vm
lc restrict
lc result
lc resume
lc ret
lc ret-type
lc retain
lc return-path
lc rev
lc rewrite-corrupted
lc right
lc ringbuf
lc ringbuf-read
lc ringbuf-write
lc ripemd160
lc riscv32
lc riscv64
lc rng-builtin
lc rng-egd
lc rng-random
lc ro
lc rounds
lc rows
lc rtc-reset-reinjection
lc rules
lc run
lc running
lc rw
lc rx
lc rxcookie
lc rxsession
lc s
lc s16
lc s32
lc s390
lc s390-pv-guest
lc s390x
lc s8
lc safe
lc sample-pages
lc sanity-check
lc sasl
lc save-vm
lc savevm-monitor-nodes
lc screendump
lc script
lc scrolllock
lc sdl
lc seal
lc second-level
lc secondary
lc secret
lc section-size
lc sections
lc sector
lc sector-num
lc sectors-count
lc semicolon
lc send-key
lc send-update
lc serial
lc serpent-128
lc serpent-192
lc serpent-256
lc server
lc server-name
lc service
lc session-file
lc set-action
lc set-eth-dst
lc set-eth-src
lc set-numa-node
lc set-speed
lc set-state
lc set-vlan-id
lc setup
lc setup-time
lc sev-device
lc sev-guest
lc sev-inject-launch-secret
lc sgx
lc sgx-epc
lc sgx1
lc sgx2
lc sh4
lc sh4eb
lc sha1
lc sha224
lc sha256
lc sha384
lc sha512
lc share
lc shared-perm
lc shift
lc shift-shift
lc show-cursor
lc shutdown
lc shutting-down
lc side
lc sig
lc signal
lc singlestep
lc size
lc skipauth
lc skipped
lc slash
lc sleep
lc slew
lc slot
lc slot-type
lc slots
lc smb
lc smbserver
lc snapshot
lc snapshot-access
lc snapshot-delete
lc snapshot-file
lc snapshot-load
lc snapshot-node-name
lc snapshot-save
lc snapshot-table
lc snapshots
lc sndbuf
lc sock
lc socket
lc socket-address
lc socket-id
lc sockets
lc source
lc sparc
lc sparc64
lc spc
lc speed
lc spice
lc spice-app
lc spiceport
lc spicevmc
lc split
lc src
lc srcport
lc ssh
lc sslverify
lc standby
lc start
lc start-server
lc start-time
lc state
lc static
lc stats
lc status
lc stdio
lc step
lc stop
lc stopped
lc str
lc stream
lc stream-name
lc string
lc stripes
lc subformat
lc subnet-prefix
lc subordinate
lc subset
lc subsystem
lc subsystem-reset
lc subsystem-vendor
lc superset
lc suspended
lc swap-opt-cmd
lc sync
lc sysrq
lc t
lc tab
lc table-size
lc tag
lc take-child-perms
lc tap
lc target
lc tbl-id
lc tcp
lc tcp-syn-count
lc telnet
lc template
lc test
lc testdev
lc tftp
lc tftp-server-name
lc third-level
lc thread-id
lc thread-pool-max
lc thread-pool-min
lc threads
lc threshold
lc throttle
lc throttle-group
lc throttle-trigger-threshold
lc tight
lc time
lc timeout
lc timer-period
lc tls
lc tls-authz
lc tls-certs
lc tls-cipher-suites
lc tls-creds
lc tls-creds-anon
lc tls-creds-psk
lc tls-creds-x509
lc tls-hostname
lc tls-none
lc tls-plain
lc tls-port
lc tls-sasl
lc tls-vnc
lc tn3270
lc to
lc toolsversion
lc top
lc top-id
lc top-node
lc total
lc total-clusters
lc total-progress
lc total-time
lc tpm-crb
lc tpm-spapr
lc tpm-tis
lc trace-event-get-state
lc trace-event-set-state
lc transaction
lc transferred
lc transform
lc transport
lc tray-open
lc tricore
lc try-mmap
lc try-poll
lc ttl-check
lc tunnel-id
lc tunnel-lport
lc twofish-128
lc twofish-192
lc twofish-256
lc tx
lc txcookie
lc txsession
lc type
lc typename
lc u
lc u16
lc u32
lc u8
lc udp
lc ultra
lc unaligned-accesses
lc unavailable
lc unavailable-features
lc undefined
lc undo
lc unicast
lc unicast-overflow
lc unicast-table
lc uninit
lc uninitialized
lc unix
lc unknown
lc unmap
lc unmapped
lc unshare-child-perms
lc unstable
lc unstable-input
lc unstable-output
lc unstarted
lc unused
lc up
lc uri
lc url
lc use-copy-range
lc user
lc username
lc utf8
lc uuid
lc v
lc v2
lc v3
lc val
lc validate-uuid
lc value
lc values
lc variants
lc vc
lc vcpu
lc vcpu-dirty-rate
lc vcpus-count
lc vde
lc vdi
lc vectors
lc vencrypt
lc vendor
lc verify-peer
lc version
lc vfio
lc vhdx
lc vhost
lc vhost-user
lc vhost-user-blk
lc vhost-vdpa
lc vhostdev
lc vhostfd
lc vhostfds
lc vhostforce
lc virtio-mem
lc virtio-pmem
lc virtual-size
lc vlan
lc vlan-id
lc vlan-table
lc vm-clock-nsec
lc vm-clock-sec
lc vm-state-size
lc vmdk
lc vmstate
lc vmstate-loaded
lc vmstate-received
lc vmstate-send
lc vmstate-size
lc vnc
lc voices
lc volume
lc volumedown
lc volumeup
lc vote-threshold
lc vpc
lc vsock
lc vvfat
lc w
lc wait
lc wait-unplug
lc waiting
lc wake
lc wakeup-suspend-support
lc watchdog
lc watchdog-set-action
lc wav
lc wctablet
lc websocket
lc wheel-down
lc wheel-left
lc wheel-right
lc wheel-up
lc width
lc win-dmp
lc window-close
lc writable
lc write
lc write-back
lc write-bandwidth
lc write-blocking
lc write-latency
lc write-threshold
lc write-through
lc write-unchanged
lc write-zeroes
lc writeback
lc writethrough
lc x
lc x-blockdev-amend
lc x-blockdev-change
lc x-blockdev-set-iothread
lc x-bps-read
lc x-bps-read-max
lc x-bps-read-max-length
lc x-bps-total
lc x-bps-total-max
lc x-bps-total-max-length
lc x-bps-write
lc x-bps-write-max
lc x-bps-write-max-length
lc x-check-cache-dropped
lc x-checkpoint-delay
lc x-colo
lc x-colo-lost-heartbeat
lc x-debug-block-dirty-bitmap-sha256
lc x-debug-query-block-graph
lc x-dirty-bitmap
lc x-exit-preconfig
lc x-ignore-shared
lc x-iops-read
lc x-iops-read-max
lc x-iops-read-max-length
lc x-iops-size
lc x-iops-total
lc x-iops-total-max
lc x-iops-total-max-length
lc x-iops-write
lc x-iops-write-max
lc x-iops-write-max-length
lc x-origin
lc x-perf
lc x-query-irq
lc x-query-jit
lc x-query-numa
lc x-query-opcount
lc x-query-profile
lc x-query-ramblock
lc x-query-rdma
lc x-query-roms
lc x-query-usb
lc x-remote-object
lc x-use-canonical-path-for-ramblock-id
lc x509-none
lc x509-plain
lc x509-sasl
lc x509-vnc
lc xbzrle
lc xbzrle-cache
lc xbzrle-cache-size
lc xen-colo-do-checkpoint
lc xen-load-devices-state
lc xen-save-devices-state
lc xen-set-global-dirty-log
lc xen-set-replication
lc xtensa
lc xtensaeb
lc xts
lc y
lc y-origin
lc yank
lc yen
lc z
lc zero
lc zero-blocks
lc zeroed-grain
lc zlib
lc zoom-to-fit
lc zstd
mc ACPIOSTInfo
mc ACPISlotType
mc COLOExitReason
mc COLOMessage
mc COLOMode
mc COLOStatus
mc DBusVMStateProperties
mc GICCapability
mc IOThreadInfo
mc JSONType
mc KVMMissingCap
mc NFSServer
mc NFSTransport
mc PCDIMMDeviceInfo
mc PCDIMMDeviceInfoWrapper
mc PCIELinkSpeed
mc PCIELinkWidth
mc PRManagerInfo
mc QAuthZListFormat
mc QAuthZListPolicy
mc QAuthZListRule
mc QCryptoBlockAmendOptions
mc QCryptoBlockAmendOptionsLUKS
mc QCryptoBlockCreateOptions
mc QCryptoBlockCreateOptionsLUKS
mc QCryptoBlockFormat
mc QCryptoBlockInfo
mc QCryptoBlockInfoBase
mc QCryptoBlockInfoLUKS
mc QCryptoBlockInfoLUKSSlot
mc QCryptoBlockLUKSKeyslotState
mc QCryptoBlockOpenOptions
mc QCryptoBlockOptionsBase
mc QCryptoBlockOptionsLUKS
mc QCryptoBlockOptionsQCow
mc QCryptoCipherAlgorithm
mc QCryptoCipherMode
mc QCryptoHashAlgorithm
mc QCryptoIVGenAlgorithm
mc QCryptoSecretFormat
mc QCryptoTLSCredsEndpoint
mc QKeyCode
mc QKeyCodeWrapper
mc QMPCapability
mc S390CrashReason
mc SGXEPCSection
mc SGXInfo
mc SMPConfiguration
mc TPMEmulatorOptions
mc TPMEmulatorOptionsWrapper
mc TPMInfo
mc TPMPassthroughOptions
mc TPMPassthroughOptionsWrapper
mc X86CPUFeatureWordInfo
mc X86CPURegister32
mc XBZRLECacheStats
mc XDbgBlockGraph
mc XDbgBlockGraphEdge
mc XDbgBlockGraphNode
mc XDbgBlockGraphNodeType
mc ac_back
mc ac_bookmarks
mc ac_forward
mc ac_home
mc ac_refresh
mc account_failed
mc account_invalid
mc add_client
mc alt_r
mc asl_compiler_id
mc asl_compiler_rev
mc avg_flush_latency_ns
mc avg_rd_latency_ns
mc avg_rd_queue_depth
mc avg_wr_latency_ns
mc avg_wr_queue_depth
mc backing_file
mc backing_file_depth
mc block_resize
mc block_set_io_throttle
mc bps_max
mc bps_max_length
mc bps_rd
mc bps_rd_max
mc bps_rd_max_length
mc bps_wr
mc bps_wr_max
mc bps_wr_max_length
mc bracket_left
mc bracket_right
mc caps_lock
mc class_info
mc client_migrate_info
mc cluster_alloc
mc cluster_alloc_bytes
mc cluster_alloc_space
mc cluster_free
mc compare_timeout
mc cor_write
mc cow_read
mc cow_write
mc ctrl_r
mc d2_5
mc detect_zeroes
mc device_add
mc device_del
mc empty_image_prepare
mc expire_password
mc expired_scan_cycle
mc failed_flush_operations
mc failed_rd_operations
mc failed_unmap_operations
mc failed_wr_operations
mc flush_latency_histogram
mc flush_operations
mc flush_to_disk
mc flush_to_os
mc flush_total_time_ns
mc grab_all
mc grave_accent
mc host_cdrom
mc host_device
mc idle_time_ns
mc interval_length
mc invalid_flush_operations
mc invalid_rd_operations
mc invalid_unmap_operations
mc invalid_wr_operations
mc io_range
mc io_uring
mc iops_max
mc iops_max_length
mc iops_rd
mc iops_rd_max
mc iops_rd_max_length
mc iops_size
mc iops_wr
mc iops_wr_max
mc iops_wr_max_length
mc irq_pin
mc known_hosts
mc kp_0
mc kp_1
mc kp_2
mc kp_3
mc kp_4
mc kp_5
mc kp_6
mc kp_7
mc kp_8
mc kp_9
mc kp_add
mc kp_comma
mc kp_decimal
mc kp_divide
mc kp_enter
mc kp_equals
mc kp_multiply
mc kp_subtract
mc l1_grow_activate_table
mc l1_grow_alloc_table
mc l1_grow_write_table
mc l1_shrink_free_l2_clusters
mc l1_shrink_write_table
mc l1_update
mc l2_alloc_cow_read
mc l2_alloc_write
mc l2_load
mc l2_update
mc l2_update_compressed
mc legacyESX
mc max_flush_latency_ns
mc max_queue_size
mc max_rd_latency_ns
mc max_wr_latency_ns
mc mem_type_64
mc memory_range
mc meta_l
mc meta_r
mc migrate_cancel
mc min_flush_latency_ns
mc min_rd_latency_ns
mc min_wr_latency_ns
mc monolithicFlat
mc monolithicSparse
mc netdev_add
mc netdev_del
mc new_state
mc notify_dev
mc num_lock
mc oem_id
mc oem_rev
mc oem_table_id
mc pci_bridge
mc prefetchable_range
mc primary_in
mc pwritev_done
mc pwritev_rmw_after_head
mc pwritev_rmw_after_tail
mc pwritev_rmw_head
mc pwritev_rmw_tail
mc pwritev_zero
mc qdev_id
mc qmp_capabilities
mc rd_bytes
mc rd_latency_histogram
mc rd_merged
mc rd_operations
mc rd_total_time_ns
mc read_aio
mc read_backing_aio
mc read_compressed
mc refblock_alloc
mc refblock_alloc_hookup
mc refblock_alloc_switch_table
mc refblock_alloc_write
mc refblock_alloc_write_blocks
mc refblock_alloc_write_table
mc refblock_load
mc refblock_update
mc refblock_update_part
mc reftable_grow
mc reftable_load
mc reftable_update
mc sasl_username
mc scroll_lock
mc secondary_in
mc secret_keyring
mc set_link
mc set_password
mc shift_r
mc streamOptimized
mc system_powerdown
mc system_reset
mc system_wakeup
mc timed_stats
mc tray_open
mc twoGbMaxExtentFlat
mc twoGbMaxExtentSparse
mc unmap_bytes
mc unmap_merged
mc unmap_operations
mc unmap_total_time_ns
mc vmstate_load
mc vmstate_save
mc vnet_hdr
mc vnet_hdr_support
mc wr_bytes
mc wr_highest_offset
mc wr_latency_histogram
mc wr_merged
mc wr_operations
mc wr_total_time_ns
mc write_aio
mc write_compressed
mc write_threshold
mc x509_dname
mc x86_64
uc ACPI_DEVICE_OST
uc BALLOON_CHANGE
uc BLOCK_EXPORT_DELETED
uc BLOCK_IMAGE_CORRUPTED
uc BLOCK_IO_ERROR
uc BLOCK_JOB_CANCELLED
uc BLOCK_JOB_COMPLETED
uc BLOCK_JOB_ERROR
uc BLOCK_JOB_PENDING
uc BLOCK_JOB_READY
uc BLOCK_WRITE_THRESHOLD
uc COLO_EXIT
uc CPU
uc DEVICE_DELETED
uc DEVICE_TRAY_MOVED
uc DEVICE_UNPLUG_GUEST_ERROR
uc DIMM
uc DUMP_COMPLETED
uc EAX
uc EBP
uc EBX
uc ECX
uc EDI
uc EDX
uc ESI
uc ESP
uc FAILOVER_NEGOTIATED
uc GUEST_CRASHLOADED
uc GUEST_PANICKED
uc JOB_STATUS_CHANGE
uc MEMORY_DEVICE_SIZE_CHANGE
uc MEMORY_FAILURE
uc MEM_UNPLUG_ERROR
uc MIGRATION
uc MIGRATION_PASS
uc NIC_RX_FILTER_CHANGED
uc POWERDOWN
uc PR_MANAGER_STATUS_CHANGED
uc QUORUM_FAILURE
uc QUORUM_REPORT_BAD
uc RDMA_GID_STATUS_CHANGED
uc RESET
uc RESUME
uc RTC_CHANGE
uc SHUTDOWN
uc SPICE_CONNECTED
uc SPICE_DISCONNECTED
uc SPICE_INITIALIZED
uc SPICE_MIGRATE_COMPLETED
uc STOP
uc SUSPEND
uc SUSPEND_DISK
uc UNPLUG_PRIMARY
uc UUID
uc VNC_CONNECTED
uc VNC_DISCONNECTED
uc VNC_INITIALIZED
uc VSERPORT_CHANGE
uc WAKEUP
uc WATCHDOGHi,
Quoted 15 lines · Markus Armbruster
Quoted 5 lines · Victor Toso
Quoted 1 line
That's true, but at least to me the trade-off feels reasonable.
I don't quite get the argument why it gets harder to find. We can simply provide the actual name as reference in the generated documentation, no?
Predictable name transformation can save me detours through documentation. Being able to guess the Go bindings just from the QMP Reference Manual can be nice: it lets me write Go code with just the QMP Reference manual in view. Being able to guess the name in the QAPI schema from the name in the Go bindings can also be nice: it lets me look it up in the QMP Reference manual without jumping through the bindings documentation.
Quoted 2 lines
Or do you envisage a Go bindings manual that fully replaces the QMP Reference Manual for developers writing Go?
When the language specifies a way of documenting the code and provides tools to generate documentation (e.g: html format) from the source code, it is common that IDEs would use this to provide documentation from a given library in a simple way.
Links break that. But we might not have a choice anyway due the fact that copying GPLv2+ could potentially be a problem to license the Go module as something else.
Quoted 9 lines
Quoted 6 lines
As Kevin already pointed out that he is not planning to work on the Alias (reference below), any other idea besides the Andrea's annotation suggestion? While this is not a blocker, I agree it would be nice to be consistent.
https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg04703.htmlLet's first try to get a handle on how many schema names are problematic.
Thanks for this. I'll reply again when I went through it (if no one else beats me to it).
Cheers,
Victor
Quoted 2,501 lines
$ git-diffscripts/qapi/expr.py +1
-0@@ -94,6 +94,7 @@ def check_name_str(name: str, info: QAPISourceInfo, source: str) -> str: """ # Reserve the entire 'q_' namespace for c_name(), and for 'q_empty' # and 'q_obj_*' implicit type names. + print("###", name) match = valid_name.match(name) if not match or c_name(name, False).startswith('q_'): raise QAPISemError(info, "%s has an invalid name" % source)$ cd bld $ python3 ../scripts/qapi-gen.py -o qapi -b ../qapi/qapi-schema.json | sort -u | awk '/^### [a-z0-9-]+$/ { print "lc", $2; next } /^### [A-Z0-9_]+$/ { print "uc", $2; next } /^### ([A-Z][a-z]+)+/ { print "cc", $2; next } { print "mc", $2 }' | sort cc Abort cc AbortWrapper cc AcpiTableOptions cc ActionCompletionMode cc AddfdInfo cc AnnounceParameters cc AudioFormat cc Audiodev cc AudiodevAlsaOptions cc AudiodevAlsaPerDirectionOptions cc AudiodevCoreaudioOptions cc AudiodevCoreaudioPerDirectionOptions cc AudiodevDriver cc AudiodevDsoundOptions cc AudiodevGenericOptions cc AudiodevJackOptions cc AudiodevJackPerDirectionOptions cc AudiodevOssOptions cc AudiodevOssPerDirectionOptions cc AudiodevPaOptions cc AudiodevPaPerDirectionOptions cc AudiodevPerDirectionOptions cc AudiodevSdlOptions cc AudiodevSdlPerDirectionOptions cc AudiodevWavOptions cc AuthZListFileProperties cc AuthZListProperties cc AuthZPAMProperties cc AuthZSimpleProperties cc BackupCommon cc BackupPerf cc BalloonInfo cc BiosAtaTranslation cc BitmapMigrationBitmapAlias cc BitmapMigrationBitmapAliasTransform cc BitmapMigrationNodeAlias cc BitmapSyncMode cc BlkdebugEvent cc BlkdebugIOType cc BlkdebugInjectErrorOptions cc BlkdebugSetStateOptions cc BlockDeviceInfo cc BlockDeviceIoStatus cc BlockDeviceStats cc BlockDeviceTimedStats cc BlockDirtyBitmap cc BlockDirtyBitmapAdd cc BlockDirtyBitmapAddWrapper cc BlockDirtyBitmapMerge cc BlockDirtyBitmapMergeWrapper cc BlockDirtyBitmapOrStr cc BlockDirtyBitmapSha256 cc BlockDirtyBitmapWrapper cc BlockDirtyInfo cc BlockErrorAction cc BlockExportInfo cc BlockExportOptions cc BlockExportOptionsFuse cc BlockExportOptionsNbd cc BlockExportOptionsNbdBase cc BlockExportOptionsVhostUserBlk cc BlockExportRemoveMode cc BlockExportType cc BlockIOThrottle cc BlockInfo cc BlockJobInfo cc BlockLatencyHistogramInfo cc BlockMeasureInfo cc BlockPermission cc BlockStats cc BlockStatsSpecific cc BlockStatsSpecificFile cc BlockStatsSpecificNvme cc BlockdevAioOptions cc BlockdevAmendOptions cc BlockdevAmendOptionsLUKS cc BlockdevAmendOptionsQcow2 cc BlockdevBackup cc BlockdevBackupWrapper cc BlockdevCacheInfo cc BlockdevCacheOptions cc BlockdevChangeReadOnlyMode cc BlockdevCreateOptions cc BlockdevCreateOptionsFile cc BlockdevCreateOptionsGluster cc BlockdevCreateOptionsLUKS cc BlockdevCreateOptionsNfs cc BlockdevCreateOptionsParallels cc BlockdevCreateOptionsQcow cc BlockdevCreateOptionsQcow2 cc BlockdevCreateOptionsQed cc BlockdevCreateOptionsRbd cc BlockdevCreateOptionsSsh cc BlockdevCreateOptionsVdi cc BlockdevCreateOptionsVhdx cc BlockdevCreateOptionsVmdk cc BlockdevCreateOptionsVpc cc BlockdevDetectZeroesOptions cc BlockdevDiscardOptions cc BlockdevDriver cc BlockdevOnError cc BlockdevOptions cc BlockdevOptionsBlkdebug cc BlockdevOptionsBlklogwrites cc BlockdevOptionsBlkreplay cc BlockdevOptionsBlkverify cc BlockdevOptionsCbw cc BlockdevOptionsCor cc BlockdevOptionsCurlBase cc BlockdevOptionsCurlFtp cc BlockdevOptionsCurlFtps cc BlockdevOptionsCurlHttp cc BlockdevOptionsCurlHttps cc BlockdevOptionsFile cc BlockdevOptionsGenericCOWFormat cc BlockdevOptionsGenericFormat cc BlockdevOptionsGluster cc BlockdevOptionsIscsi cc BlockdevOptionsLUKS cc BlockdevOptionsNVMe cc BlockdevOptionsNbd cc BlockdevOptionsNfs cc BlockdevOptionsNull cc BlockdevOptionsPreallocate cc BlockdevOptionsQcow cc BlockdevOptionsQcow2 cc BlockdevOptionsQuorum cc BlockdevOptionsRaw cc BlockdevOptionsRbd cc BlockdevOptionsReplication cc BlockdevOptionsSsh cc BlockdevOptionsThrottle cc BlockdevOptionsVVFAT cc BlockdevQcow2Encryption cc BlockdevQcow2EncryptionFormat cc BlockdevQcow2Version cc BlockdevQcowEncryption cc BlockdevQcowEncryptionFormat cc BlockdevRef cc BlockdevRefOrNull cc BlockdevSnapshot cc BlockdevSnapshotInternal cc BlockdevSnapshotInternalWrapper cc BlockdevSnapshotSync cc BlockdevSnapshotSyncWrapper cc BlockdevSnapshotWrapper cc BlockdevVhdxSubformat cc BlockdevVmdkAdapterType cc BlockdevVmdkSubformat cc BlockdevVpcSubformat cc CanHostSocketcanProperties cc ChardevBackend cc ChardevBackendInfo cc ChardevBackendKind cc ChardevCommon cc ChardevCommonWrapper cc ChardevDBus cc ChardevDBusWrapper cc ChardevFile cc ChardevFileWrapper cc ChardevHostdev cc ChardevHostdevWrapper cc ChardevInfo cc ChardevMux cc ChardevMuxWrapper cc ChardevQemuVDAgent cc ChardevQemuVDAgentWrapper cc ChardevReturn cc ChardevRingbuf cc ChardevRingbufWrapper cc ChardevSocket cc ChardevSocketWrapper cc ChardevSpiceChannel cc ChardevSpiceChannelWrapper cc ChardevSpicePort cc ChardevSpicePortWrapper cc ChardevStdio cc ChardevStdioWrapper cc ChardevUdp cc ChardevUdpWrapper cc ChardevVC cc ChardevVCWrapper cc ColoCompareProperties cc CommandInfo cc CommandLineOptionInfo cc CommandLineParameterInfo cc CommandLineParameterType cc CommandNotFound cc CompatPolicy cc CompatPolicyInput cc CompatPolicyOutput cc CompressionStats cc CpuDefinitionInfo cc CpuInfoFast cc CpuInfoS390 cc CpuInstanceProperties cc CpuModelBaselineInfo cc CpuModelCompareInfo cc CpuModelCompareResult cc CpuModelExpansionInfo cc CpuModelExpansionType cc CpuModelInfo cc CpuS390State cc CryptodevBackendProperties cc CryptodevVhostUserProperties cc CurrentMachineParams cc DataFormat cc DeviceNotActive cc DeviceNotFound cc DirtyRateInfo cc DirtyRateMeasureMode cc DirtyRateStatus cc DirtyRateVcpu cc DisplayCocoa cc DisplayCurses cc DisplayDBus cc DisplayEGLHeadless cc DisplayGLMode cc DisplayGTK cc DisplayOptions cc DisplayProtocol cc DisplayReloadOptions cc DisplayReloadOptionsVNC cc DisplayReloadType cc DisplayType cc DisplayUpdateOptions cc DisplayUpdateOptionsVNC cc DisplayUpdateType cc DriveBackup cc DriveBackupWrapper cc DriveMirror cc DummyForceArrays cc DumpGuestMemoryCapability cc DumpGuestMemoryFormat cc DumpQueryResult cc DumpStatus cc EventLoopBaseProperties cc ExpirePasswordOptions cc ExpirePasswordOptionsVnc cc FailoverStatus cc FdsetFdInfo cc FdsetInfo cc FilterBufferProperties cc FilterDumpProperties cc FilterMirrorProperties cc FilterRedirectorProperties cc FilterRewriterProperties cc FloppyDriveType cc FuseExportAllowOther cc GenericError cc GrabToggleKeys cc GuestPanicAction cc GuestPanicInformation cc GuestPanicInformationHyperV cc GuestPanicInformationS390 cc GuestPanicInformationType cc GuidInfo cc HmatCacheAssociativity cc HmatCacheWritePolicy cc HmatLBDataType cc HmatLBMemoryHierarchy cc HostMemPolicy cc HotpluggableCPU cc HumanReadableText cc ImageCheck cc ImageFormat cc ImageInfo cc ImageInfoSpecific cc ImageInfoSpecificKind cc ImageInfoSpecificLUKSWrapper cc ImageInfoSpecificQCow2 cc ImageInfoSpecificQCow2Encryption cc ImageInfoSpecificQCow2EncryptionBase cc ImageInfoSpecificQCow2Wrapper cc ImageInfoSpecificRbd cc ImageInfoSpecificRbdWrapper cc ImageInfoSpecificVmdk cc ImageInfoSpecificVmdkWrapper cc InetSocketAddress cc InetSocketAddressBase cc InetSocketAddressWrapper cc InputAxis cc InputBarrierProperties cc InputBtnEvent cc InputBtnEventWrapper cc InputButton cc InputEvent cc InputEventKind cc InputKeyEvent cc InputKeyEventWrapper cc InputLinuxProperties cc InputMoveEvent cc InputMoveEventWrapper cc IntWrapper cc IoOperationType cc IothreadProperties cc IscsiHeaderDigest cc IscsiTransport cc JobInfo cc JobStatus cc JobType cc JobVerb cc KeyValue cc KeyValueKind cc KvmInfo cc LostTickPolicy cc MachineInfo cc MainLoopProperties cc MapEntry cc Memdev cc MemoryBackendEpcProperties cc MemoryBackendFileProperties cc MemoryBackendMemfdProperties cc MemoryBackendProperties cc MemoryDeviceInfo cc MemoryDeviceInfoKind cc MemoryFailureAction cc MemoryFailureFlags cc MemoryFailureRecipient cc MemoryInfo cc MigrateSetParameters cc MigrationCapability cc MigrationCapabilityStatus cc MigrationInfo cc MigrationParameter cc MigrationParameters cc MigrationStats cc MigrationStatus cc MirrorCopyMode cc MirrorSyncMode cc MonitorMode cc MonitorOptions cc MouseInfo cc MultiFDCompression cc NameInfo cc NbdServerAddOptions cc NbdServerOptions cc NetClientDriver cc NetFilterDirection cc NetLegacyNicOptions cc Netdev cc NetdevBridgeOptions cc NetdevHubPortOptions cc NetdevL2TPv3Options cc NetdevNetmapOptions cc NetdevSocketOptions cc NetdevTapOptions cc NetdevUserOptions cc NetdevVdeOptions cc NetdevVhostUserOptions cc NetdevVhostVDPAOptions cc NetfilterInsert cc NetfilterProperties cc NetworkAddressFamily cc NewImageMode cc NumaCpuOptions cc NumaDistOptions cc NumaHmatCacheOptions cc NumaHmatLBOptions cc NumaNodeOptions cc NumaOptions cc NumaOptionsType cc ObjectOptions cc ObjectPropertyInfo cc ObjectType cc ObjectTypeInfo cc OffAutoPCIBAR cc OnOffAuto cc OnOffSplit cc PanicAction cc PciBridgeInfo cc PciBusInfo cc PciDeviceClass cc PciDeviceId cc PciDeviceInfo cc PciInfo cc PciMemoryRange cc PciMemoryRegion cc PrManagerHelperProperties cc PreallocMode cc QapiErrorClass cc Qcow2BitmapInfo cc Qcow2BitmapInfoFlags cc Qcow2CompressionType cc Qcow2OverlapCheckFlags cc Qcow2OverlapCheckMode cc Qcow2OverlapChecks cc QtestProperties cc QuorumOpType cc QuorumReadPattern cc RbdAuthMode cc RbdEncryptionCreateOptions cc RbdEncryptionCreateOptionsLUKS cc RbdEncryptionCreateOptionsLUKS2 cc RbdEncryptionCreateOptionsLUKSBase cc RbdEncryptionOptions cc RbdEncryptionOptionsLUKS cc RbdEncryptionOptionsLUKS2 cc RbdEncryptionOptionsLUKSBase cc RbdImageEncryptionFormat cc RebootAction cc RemoteObjectProperties cc ReplayInfo cc ReplayMode cc ReplicationMode cc ReplicationStatus cc RngEgdProperties cc RngProperties cc RngRandomProperties cc RockerOfDpaFlow cc RockerOfDpaFlowAction cc RockerOfDpaFlowKey cc RockerOfDpaFlowMask cc RockerOfDpaGroup cc RockerPort cc RockerPortAutoneg cc RockerPortDuplex cc RockerSwitch cc RunState cc RxFilterInfo cc RxState cc SchemaInfo cc SchemaInfoAlternate cc SchemaInfoAlternateMember cc SchemaInfoArray cc SchemaInfoBuiltin cc SchemaInfoCommand cc SchemaInfoEnum cc SchemaInfoEnumMember cc SchemaInfoEvent cc SchemaInfoObject cc SchemaInfoObjectMember cc SchemaInfoObjectVariant cc SchemaMetaType cc SecretCommonProperties cc SecretKeyringProperties cc SecretProperties cc SetPasswordAction cc SetPasswordOptions cc SetPasswordOptionsVnc cc SevAttestationReport cc SevCapability cc SevGuestProperties cc SevInfo cc SevLaunchMeasureInfo cc SevState cc SgxEPC cc SgxEPCDeviceInfo cc SgxEPCDeviceInfoWrapper cc SgxEPCProperties cc ShutdownAction cc ShutdownCause cc SmbiosEntryPointType cc SnapshotInfo cc SocketAddress cc SocketAddressLegacy cc SocketAddressType cc SpiceBasicInfo cc SpiceChannel cc SpiceInfo cc SpiceQueryMouseMode cc SpiceServerInfo cc SshHostKeyCheck cc SshHostKeyCheckHashType cc SshHostKeyCheckMode cc SshHostKeyHash cc StatusInfo cc StrOrNull cc String cc StringWrapper cc SysEmuTarget cc TargetInfo cc ThrottleGroupProperties cc ThrottleLimits cc TlsCredsAnonProperties cc TlsCredsProperties cc TlsCredsPskProperties cc TlsCredsX509Properties cc TpmModel cc TpmType cc TpmTypeOptions cc TraceEventInfo cc TraceEventState cc TransactionAction cc TransactionActionKind cc TransactionProperties cc UnixSocketAddress cc UnixSocketAddressWrapper cc UuidInfo cc VersionInfo cc VersionTriple cc VfioStats cc VirtioMEMDeviceInfo cc VirtioMEMDeviceInfoWrapper cc VirtioPMEMDeviceInfo cc VirtioPMEMDeviceInfoWrapper cc VncBasicInfo cc VncClientInfo cc VncInfo cc VncInfo2 cc VncPrimaryAuth cc VncServerInfo cc VncServerInfo2 cc VncVencryptSubAuth cc VsockSocketAddress cc VsockSocketAddressWrapper cc WatchdogAction cc YankInstance cc YankInstanceBlockNode cc YankInstanceChardev cc YankInstanceType lc a lc aarch64 lc abort lc aborting lc abs lc absolute lc absolute-paths lc abstract lc accept lc access-bandwidth lc access-latency lc action lc action-required lc actions lc active lc active-l1 lc active-l2 lc actual lc actual-size lc adapter-type lc add-fd lc addr lc address lc addresses lc aes lc aes-128 lc aes-192 lc aes-256 lc again lc aio lc aio-max-batch lc alias lc alias-of lc align lc aligned-accesses lc all lc allocated-clusters lc allocation-depth lc allow lc allow-oob lc allow-other lc allow-write-only-overlay lc alpha lc alsa lc alt lc alt-alt lc alternate lc always lc amend lc amount-exceeded lc announce-initial lc announce-max lc announce-rounds lc announce-self lc announce-step lc api-major lc api-minor lc apostrophe lc append lc arch lc arg-type lc arg1 lc arg2 lc arg3 lc arg4 lc arg5 lc arm lc array lc associativity lc asterisk lc audiodev lc audiomute lc audionext lc audioplay lc audioprev lc audiostop lc auth lc auth-client-required lc authz-list lc authz-listfile lc authz-pam lc authz-simple lc auto lc auto-converge lc auto-dismiss lc auto-finalize lc auto-read-only lc autoneg lc avr lc axis lc b lc backend lc background lc background-snapshot lc backing lc backing-file lc backing-filename lc backing-filename-format lc backing-fmt lc backing-image lc backslash lc backspace lc backup lc balloon lc bandwidth lc bar lc bar0 lc bar1 lc bar2 lc bar3 lc bar4 lc bar5 lc base lc base-memory lc base-node lc base64 lc before lc begin lc behind lc bind lc bins lc bitmap lc bitmap-directory lc bitmap-mode lc bitmaps lc blk lc blkdebug lc blklogwrites lc blkreplay lc blkverify lc block lc block-backend lc block-bitmap-mapping lc block-commit lc block-dirty-bitmap-add lc block-dirty-bitmap-clear lc block-dirty-bitmap-disable lc block-dirty-bitmap-enable lc block-dirty-bitmap-merge lc block-dirty-bitmap-remove lc block-driver lc block-export-add lc block-export-del lc block-incremental lc block-job lc block-job-cancel lc block-job-complete lc block-job-dismiss lc block-job-finalize lc block-job-pause lc block-job-resume lc block-job-set-speed lc block-latency-histogram-set lc block-node lc block-set-write-threshold lc block-size lc block-state-zero lc block-status lc block-stream lc blockdev-add lc blockdev-backup lc blockdev-change-medium lc blockdev-close-tray lc blockdev-create lc blockdev-del lc blockdev-insert-medium lc blockdev-mirror lc blockdev-open-tray lc blockdev-remove-medium lc blockdev-reopen lc blockdev-snapshot lc blockdev-snapshot-delete-internal-sync lc blockdev-snapshot-internal-sync lc blockdev-snapshot-sync lc blocked-reasons lc bochs lc boolean lc bootfile lc bottom lc boundaries lc boundaries-flush lc boundaries-read lc boundaries-write lc bps lc bps-read lc bps-read-max lc bps-read-max-length lc bps-total lc bps-total-max lc bps-total-max-length lc bps-write lc bps-write-max lc bps-write-max-length lc br lc braille lc bridge lc broadcast-allowed lc btn lc buf-size lc buffer-count lc buffer-length lc build-id lc builtin lc bus lc buslogic lc busy lc busy-rate lc button lc bytes lc c lc cache lc cache-clean-interval lc cache-miss lc cache-miss-rate lc cache-size lc cached lc calc-dirty-rate lc calc-time lc calculator lc can-bus lc can-host-socketcan lc canbus lc cancel lc cancel-path lc cancelled lc cancelling lc capabilities lc capability lc case lc cast5-128 lc cbc lc cbitpos lc cephx lc cert-chain lc cert-subject lc change-backing-file lc change-vnc-password lc channel-id lc channel-type lc channels lc chardev lc chardev-add lc chardev-change lc chardev-remove lc chardev-send-break lc charset lc check-errors lc check-stop lc checkpoint-ready lc checkpoint-reply lc checkpoint-request lc child lc children lc cid lc cipher-alg lc cipher-mode lc class lc client lc client-name lc clients lc clipboard lc cloop lc closefd lc cluster-id lc cluster-size lc clusters lc cocoa lc colo lc colo-compare lc cols lc comma lc command lc command-line lc commit lc compat lc compiled-version lc complete lc completed lc completion-errors lc completion-mode lc complex lc compose lc compress lc compress-level lc compress-threads lc compress-wait-thread lc compressed lc compressed-clusters lc compressed-size lc compression lc compression-rate lc compression-type lc computer lc concluded lc conf lc config lc connect lc connect-ports lc connected lc connection-id lc consistent-read lc console lc constant lc cont lc control lc cookie lc cookie-secret lc cookie64 lc copy lc copy-before-write lc copy-mode lc copy-on-read lc core lc core-id lc coreaudio lc cores lc corrupt lc corruptions lc corruptions-fixed lc count lc counter lc cpu lc cpu-index lc cpu-max lc cpu-state lc cpu-throttle-increment lc cpu-throttle-initial lc cpu-throttle-percentage lc cpu-throttle-tailslow lc cpu0-id lc cpuid-input-eax lc cpuid-input-ecx lc cpuid-register lc cpus lc crash lc crc32c lc crc32c-none lc create lc create-type lc created lc cris lc cryptodev-backend lc cryptodev-backend-builtin lc cryptodev-vhost-user lc ctr lc ctrl lc ctrl-ctrl lc ctrl-scrolllock lc current lc current-progress lc curses lc cut lc d lc d0 lc d1 lc d12 lc d120 lc d144 lc d16 lc d2 lc d288 lc d3 lc d32 lc d3des lc d4 lc d5 lc d6 lc d64 lc d7 lc d8 lc d9 lc data lc data-file lc data-file-raw lc data-type lc date-nsec lc date-sec lc dbus lc dbus-vmstate lc debug lc decompress-threads lc default lc default-cpu-type lc default-ram-id lc default-value lc definition lc delay lc delete lc deny lc deprecated lc deprecated-input lc deprecated-output lc depth lc des lc desc lc description lc detach lc detect-zeroes lc dev lc device lc device-id lc device-list-properties lc devices lc devid lc devname lc dh-cert-file lc dhcpstart lc die-id lc dies lc dimm lc dir lc direct lc dirty-bitmap lc dirty-bitmaps lc dirty-flag lc dirty-pages-rate lc dirty-rate lc dirty-ring lc dirty-sync-count lc disabled lc disabled-wait lc discard lc discard-bytes-ok lc discard-data lc discard-nb-failed lc discard-nb-ok lc disconnect lc disk lc dismiss lc display lc display-reload lc display-update lc dist lc dmg lc dns lc dnssearch lc domainname lc dot lc down lc downscript lc downtime lc downtime-bytes lc downtime-limit lc drive-backup lc drive-mirror lc driver lc driver-specific lc drop-cache lc drv lc dsound lc dsp-policy lc dst lc dstport lc dump lc dump-guest-memory lc dump-skeys lc duplex lc duplicate lc dynamic lc dynamic-auto-read-only lc e lc ecb lc edges lc egl-headless lc eject lc element-type lc elf lc emulated lc emulator lc enable lc enabled lc encoding-rate lc encrypt lc encrypted lc encryption-format lc end lc endpoint lc enospc lc enum lc equal lc errno lc error lc error-desc lc es lc esc lc essiv lc eth-dst lc eth-src lc eth-type lc evdev lc event lc events lc exact lc exact-name lc exclusive lc existing lc expected-downtime lc export lc extended-l2 lc extent-size-hint lc extents lc external lc extint-loop lc extra lc f lc f1 lc f10 lc f11 lc f12 lc f2 lc f3 lc f32 lc f4 lc f5 lc f6 lc f7 lc f8 lc f9 lc fail lc failed lc failover lc falloc lc family lc fat-type lc fatal lc fd lc fdname lc fds lc fdset-id lc features lc fifo lc file lc filename lc filter-buffer lc filter-dump lc filter-mirror lc filter-node-name lc filter-redirector lc filter-replay lc filter-rewriter lc finalize lc find lc finish-migrate lc first-level lc fixed lc fixed-iothread lc fixed-settings lc flags lc flat lc flc lc floppy lc flush lc force lc force-share lc force-size lc format lc format-specific lc formats lc fqdn lc fragmented-clusters lc frequency lc front lc frontend-open lc ftp lc ftps lc full lc full-backing-filename lc full-grab lc full-screen lc fully-allocated lc function lc fuse lc g lc getfd lc gid-status lc gl lc glob lc gluster lc goto-tbl lc gpa lc grab-on-hover lc grab-toggle lc granularity lc group lc group-id lc group-ids lc grouped lc growable lc gtk lc guest lc guest-panic lc guest-panicked lc guest-reset lc guest-shutdown lc guestfwd lc guid lc h lc half lc handle lc hard lc hash lc hash-alg lc head lc header-digest lc height lc help lc helper lc henkan lc hide lc hierarchy lc hiragana lc hits lc hmat-cache lc hmat-lb lc hold-time lc home lc host lc host-error lc host-key-check lc host-nodes lc host-qmp-quit lc host-qmp-system-reset lc host-signal lc host-ui lc hostfwd lc hostname lc hotpluggable lc hotpluggable-cpus lc hotplugged lc hppa lc http lc https lc hubid lc hubport lc hugetlb lc hugetlbsize lc human-monitor-command lc human-readable-text lc hwversion lc hyper-v lc hypervisor lc i lc i386 lc icount lc id lc id-list lc ide lc identical lc identity lc if lc ifname lc ignore lc ignore-unavailable lc image lc image-end-offset lc image-node-name lc immediately lc implements lc in lc in-pport lc in-use lc inactive lc inactive-l1 lc inactive-l2 lc inc lc incompatible lc inconsistent lc incremental lc indev lc index lc individual lc inet lc info lc initial lc initiator lc initiator-name lc inject lc inject-error lc inject-nmi lc inmigrate lc input-barrier lc input-linux lc input-send-event lc insert lc inserted lc instances lc int lc interface-id lc interfaces lc interleave lc internal-error lc interval lc io-error lc io-status lc iops lc iops-read lc iops-read-max lc iops-read-max-length lc iops-size lc iops-total lc iops-total-max lc iops-total-max-length lc iops-write lc iops-write-max lc iops-write-max-length lc iothread lc iotype lc ip lc ip-dst lc ip-proto lc ip-tos lc ipv4 lc ipv6 lc ipv6-dns lc ipv6-host lc ipv6-prefix lc ipv6-prefixlen lc irq lc is-default lc iscsi lc iser lc iter-time lc iters lc iv lc ivgen-alg lc ivgen-hash-alg lc j lc jack lc job-cancel lc job-complete lc job-dismiss lc job-finalize lc job-id lc job-pause lc job-resume lc json-cli lc json-cli-hotplug lc json-type lc k lc katakanahiragana lc kdump-lzo lc kdump-snappy lc kdump-zlib lc keep lc keep-alive lc kernel lc kernel-hashes lc key lc key-offset lc key-secret lc keyid lc keys lc keyslot lc l lc l2-cache-entry-size lc l2-cache-size lc l2tpv3 lc label lc lang1 lc lang2 lc large lc last-mode lc late-block-activate lc latency lc latency-ns lc launch-secret lc launch-update lc lazy-refcounts lc lba lc leaks lc leaks-fixed lc left lc left-command-key lc len lc length lc less lc level lc lf lc limit lc limits lc line lc link-up lc listen lc live lc load lc loaded lc local lc localaddr lc location lc locked lc locking lc log lc log-append lc log-sector-size lc log-size lc log-super-update-interval lc logappend lc logfile lc logical-block-size lc lsilogic lc luks lc luks2 lc lun lc m lc m68k lc macaddr lc mail lc main-header lc main-loop lc main-mac lc major lc mask lc master-key-iters lc match lc max lc max-bandwidth lc max-chunk lc max-connections lc max-cpu-throttle lc max-discard lc max-postcopy-bandwidth lc max-size lc max-transfer lc max-workers lc max-write-zero lc maxcpus lc maxlen lc mbps lc mcast lc md5 lc measured lc measuring lc mediaselect lc mem lc mem-path lc memaddr lc members lc memdev lc memory lc memory-backend-epc lc memory-backend-file lc memory-backend-memfd lc memory-backend-ram lc memsave lc menu lc merge lc meta-meta lc meta-type lc metadata lc micro lc microblaze lc microblazeel lc middle lc migrate lc migrate-continue lc migrate-incoming lc migrate-pause lc migrate-recover lc migrate-set-capabilities lc migrate-set-parameters lc migrate-start-postcopy lc migrated lc migration lc migration-safe lc minor lc minus lc mips lc mips64 lc mips64el lc mipsel lc mirror lc mixing-engine lc mnonce lc mode lc model lc modela lc modelb lc mountpoint lc mouse lc mouse-mode lc mptcp lc msg lc msmouse lc muhenkan lc multicast lc multicast-overflow lc multicast-table lc multifd lc multifd-bytes lc multifd-channels lc multifd-compression lc multifd-zlib-level lc multifd-zstd-level lc mux lc n lc name lc namespace lc native lc nbd lc nbd-server-add lc nbd-server-remove lc nbd-server-start lc nbd-server-stop lc net lc netdev lc netmap lc never lc new-secret lc new-vlan-id lc nfs lc nic lc nios2 lc no-flush lc nocow lc node lc node-id lc node-name lc nodeid lc nodelay lc nodes lc none lc none-crc32c lc normal lc normal-bytes lc nospace lc null lc null-aio lc null-co lc num-queues lc numa-mem-supported lc number lc numeric lc nvdimm lc nvme lc o lc object lc object-add lc object-del lc off lc offset lc ok lc old-secret lc on lc on-error lc on-source-error lc on-success lc on-target-error lc once lc oob lc opaque lc open lc open-timeout lc opened lc operating lc operation lc opint-loop lc opt-discard lc opt-write-zero lc option lc options lc or1k lc oss lc out lc out-pport lc outdev lc overflow lc overlap-check lc overlay lc p lc p2p lc pa lc package lc packet-header lc page-cache-size lc page-sampling lc page-size lc pages lc pages-per-second lc paging lc panic lc parallel lc parallels lc parameters lc parent lc parent-cid lc pass lc pass-discard-other lc pass-discard-request lc pass-discard-snapshot lc passthrough lc password lc password-secret lc passwordid lc paste lc path lc pause lc pause-before-switchover lc paused lc payload-offset lc pdh lc pef-guest lc pending lc period-length lc perm lc persistent lc pgdn lc pgmint-loop lc pgup lc pincounter lc pipe lc plain lc plain64 lc play lc plugged-memory lc pmem lc pmemsave lc png lc policy lc poll-grow lc poll-max-ns lc poll-shrink lc poll-us lc pool lc pop-vlan lc port lc portal lc ports lc position lc postcopy-active lc postcopy-blocktime lc postcopy-bytes lc postcopy-paused lc postcopy-ram lc postcopy-recover lc postcopy-requests lc postcopy-vcpu-blocktime lc postmigrate lc power lc poweroff lc ppc lc ppc64 lc ppm lc pport lc pr-manager lc pr-manager-helper lc pre-switchover lc prealloc lc prealloc-align lc prealloc-size lc prealloc-threads lc preallocate lc preallocation lc precopy-bytes lc preferred lc prefetch lc prelaunch lc present lc pretty lc primary lc print lc priority lc processing lc promiscuous lc properties lc property lc props lc protocol lc proxy-password-secret lc proxy-username lc psw-addr lc psw-mask lc pty lc pwritev lc q lc qcode lc qcow lc qcow2 lc qdev lc qed lc qemu lc qemu-vdagent lc qom-get lc qom-list lc qom-list-properties lc qom-list-types lc qom-path lc qom-set lc qom-type lc qtest lc query-acpi-ospm-status lc query-balloon lc query-block lc query-block-exports lc query-block-jobs lc query-blockstats lc query-chardev lc query-chardev-backends lc query-colo-status lc query-command-line-options lc query-commands lc query-cpu-definitions lc query-cpu-model-baseline lc query-cpu-model-comparison lc query-cpu-model-expansion lc query-cpus-fast lc query-current-machine lc query-dirty-rate lc query-display-options lc query-dump lc query-dump-guest-memory-capability lc query-fdsets lc query-gic-capabilities lc query-hotpluggable-cpus lc query-iothreads lc query-jobs lc query-kvm lc query-machines lc query-memdev lc query-memory-devices lc query-memory-size-summary lc query-mice lc query-migrate lc query-migrate-capabilities lc query-migrate-parameters lc query-name lc query-named-block-nodes lc query-nodes lc query-pci lc query-pr-managers lc query-qmp-schema lc query-replay lc query-rocker lc query-rocker-of-dpa-flows lc query-rocker-of-dpa-groups lc query-rocker-ports lc query-rx-filter lc query-sev lc query-sev-attestation-report lc query-sev-capabilities lc query-sev-launch-measure lc query-sgx lc query-sgx-capabilities lc query-spice lc query-status lc query-target lc query-tpm lc query-tpm-models lc query-tpm-types lc query-uuid lc query-version lc query-vm-generation-id lc query-vnc lc query-vnc-servers lc query-xen-replication-status lc query-yank lc queue lc queues lc quit lc quorum lc r lc ra2 lc ra2ne lc ram lc raw lc rbd lc rdma-pin-all lc read lc read-bandwidth lc read-latency lc read-only lc read-only-mode lc read-pattern lc read-write lc read-zeroes lc readahead lc readahead-size lc readline lc readonly lc ready lc reason lc reboot lc receive-update lc rechs lc recipient lc reconnect lc reconnect-delay lc record lc recording lc recursive lc reduced-phys-bits lc refcount-bits lc refcount-block lc refcount-cache-size lc refcount-table lc reference lc refresh lc regions lc reject lc rel lc relaunch lc release-ram lc remaining lc remote lc removable lc remove-fd lc rendernode lc repeat lc replaces lc replay-break lc replay-delete-break lc replay-seek lc replication lc report lc request lc requested-size lc require lc required lc reserve lc reset lc resize lc responsible-properties lc restore-vm lc restrict lc result lc resume lc ret lc ret-type lc retain lc return-path lc rev lc rewrite-corrupted lc right lc ringbuf lc ringbuf-read lc ringbuf-write lc ripemd160 lc riscv32 lc riscv64 lc rng-builtin lc rng-egd lc rng-random lc ro lc rounds lc rows lc rtc-reset-reinjection lc rules lc run lc running lc rw lc rx lc rxcookie lc rxsession lc s lc s16 lc s32 lc s390 lc s390-pv-guest lc s390x lc s8 lc safe lc sample-pages lc sanity-check lc sasl lc save-vm lc savevm-monitor-nodes lc screendump lc script lc scrolllock lc sdl lc seal lc second-level lc secondary lc secret lc section-size lc sections lc sector lc sector-num lc sectors-count lc semicolon lc send-key lc send-update lc serial lc serpent-128 lc serpent-192 lc serpent-256 lc server lc server-name lc service lc session-file lc set-action lc set-eth-dst lc set-eth-src lc set-numa-node lc set-speed lc set-state lc set-vlan-id lc setup lc setup-time lc sev-device lc sev-guest lc sev-inject-launch-secret lc sgx lc sgx-epc lc sgx1 lc sgx2 lc sh4 lc sh4eb lc sha1 lc sha224 lc sha256 lc sha384 lc sha512 lc share lc shared-perm lc shift lc shift-shift lc show-cursor lc shutdown lc shutting-down lc side lc sig lc signal lc singlestep lc size lc skipauth lc skipped lc slash lc sleep lc slew lc slot lc slot-type lc slots lc smb lc smbserver lc snapshot lc snapshot-access lc snapshot-delete lc snapshot-file lc snapshot-load lc snapshot-node-name lc snapshot-save lc snapshot-table lc snapshots lc sndbuf lc sock lc socket lc socket-address lc socket-id lc sockets lc source lc sparc lc sparc64 lc spc lc speed lc spice lc spice-app lc spiceport lc spicevmc lc split lc src lc srcport lc ssh lc sslverify lc standby lc start lc start-server lc start-time lc state lc static lc stats lc status lc stdio lc step lc stop lc stopped lc str lc stream lc stream-name lc string lc stripes lc subformat lc subnet-prefix lc subordinate lc subset lc subsystem lc subsystem-reset lc subsystem-vendor lc superset lc suspended lc swap-opt-cmd lc sync lc sysrq lc t lc tab lc table-size lc tag lc take-child-perms lc tap lc target lc tbl-id lc tcp lc tcp-syn-count lc telnet lc template lc test lc testdev lc tftp lc tftp-server-name lc third-level lc thread-id lc thread-pool-max lc thread-pool-min lc threads lc threshold lc throttle lc throttle-group lc throttle-trigger-threshold lc tight lc time lc timeout lc timer-period lc tls lc tls-authz lc tls-certs lc tls-cipher-suites lc tls-creds lc tls-creds-anon lc tls-creds-psk lc tls-creds-x509 lc tls-hostname lc tls-none lc tls-plain lc tls-port lc tls-sasl lc tls-vnc lc tn3270 lc to lc toolsversion lc top lc top-id lc top-node lc total lc total-clusters lc total-progress lc total-time lc tpm-crb lc tpm-spapr lc tpm-tis lc trace-event-get-state lc trace-event-set-state lc transaction lc transferred lc transform lc transport lc tray-open lc tricore lc try-mmap lc try-poll lc ttl-check lc tunnel-id lc tunnel-lport lc twofish-128 lc twofish-192 lc twofish-256 lc tx lc txcookie lc txsession lc type lc typename lc u lc u16 lc u32 lc u8 lc udp lc ultra lc unaligned-accesses lc unavailable lc unavailable-features lc undefined lc undo lc unicast lc unicast-overflow lc unicast-table lc uninit lc uninitialized lc unix lc unknown lc unmap lc unmapped lc unshare-child-perms lc unstable lc unstable-input lc unstable-output lc unstarted lc unused lc up lc uri lc url lc use-copy-range lc user lc username lc utf8 lc uuid lc v lc v2 lc v3 lc val lc validate-uuid lc value lc values lc variants lc vc lc vcpu lc vcpu-dirty-rate lc vcpus-count lc vde lc vdi lc vectors lc vencrypt lc vendor lc verify-peer lc version lc vfio lc vhdx lc vhost lc vhost-user lc vhost-user-blk lc vhost-vdpa lc vhostdev lc vhostfd lc vhostfds lc vhostforce lc virtio-mem lc virtio-pmem lc virtual-size lc vlan lc vlan-id lc vlan-table lc vm-clock-nsec lc vm-clock-sec lc vm-state-size lc vmdk lc vmstate lc vmstate-loaded lc vmstate-received lc vmstate-send lc vmstate-size lc vnc lc voices lc volume lc volumedown lc volumeup lc vote-threshold lc vpc lc vsock lc vvfat lc w lc wait lc wait-unplug lc waiting lc wake lc wakeup-suspend-support lc watchdog lc watchdog-set-action lc wav lc wctablet lc websocket lc wheel-down lc wheel-left lc wheel-right lc wheel-up lc width lc win-dmp lc window-close lc writable lc write lc write-back lc write-bandwidth lc write-blocking lc write-latency lc write-threshold lc write-through lc write-unchanged lc write-zeroes lc writeback lc writethrough lc x lc x-blockdev-amend lc x-blockdev-change lc x-blockdev-set-iothread lc x-bps-read lc x-bps-read-max lc x-bps-read-max-length lc x-bps-total lc x-bps-total-max lc x-bps-total-max-length lc x-bps-write lc x-bps-write-max lc x-bps-write-max-length lc x-check-cache-dropped lc x-checkpoint-delay lc x-colo lc x-colo-lost-heartbeat lc x-debug-block-dirty-bitmap-sha256 lc x-debug-query-block-graph lc x-dirty-bitmap lc x-exit-preconfig lc x-ignore-shared lc x-iops-read lc x-iops-read-max lc x-iops-read-max-length lc x-iops-size lc x-iops-total lc x-iops-total-max lc x-iops-total-max-length lc x-iops-write lc x-iops-write-max lc x-iops-write-max-length lc x-origin lc x-perf lc x-query-irq lc x-query-jit lc x-query-numa lc x-query-opcount lc x-query-profile lc x-query-ramblock lc x-query-rdma lc x-query-roms lc x-query-usb lc x-remote-object lc x-use-canonical-path-for-ramblock-id lc x509-none lc x509-plain lc x509-sasl lc x509-vnc lc xbzrle lc xbzrle-cache lc xbzrle-cache-size lc xen-colo-do-checkpoint lc xen-load-devices-state lc xen-save-devices-state lc xen-set-global-dirty-log lc xen-set-replication lc xtensa lc xtensaeb lc xts lc y lc y-origin lc yank lc yen lc z lc zero lc zero-blocks lc zeroed-grain lc zlib lc zoom-to-fit lc zstd mc ACPIOSTInfo mc ACPISlotType mc COLOExitReason mc COLOMessage mc COLOMode mc COLOStatus mc DBusVMStateProperties mc GICCapability mc IOThreadInfo mc JSONType mc KVMMissingCap mc NFSServer mc NFSTransport mc PCDIMMDeviceInfo mc PCDIMMDeviceInfoWrapper mc PCIELinkSpeed mc PCIELinkWidth mc PRManagerInfo mc QAuthZListFormat mc QAuthZListPolicy mc QAuthZListRule mc QCryptoBlockAmendOptions mc QCryptoBlockAmendOptionsLUKS mc QCryptoBlockCreateOptions mc QCryptoBlockCreateOptionsLUKS mc QCryptoBlockFormat mc QCryptoBlockInfo mc QCryptoBlockInfoBase mc QCryptoBlockInfoLUKS mc QCryptoBlockInfoLUKSSlot mc QCryptoBlockLUKSKeyslotState mc QCryptoBlockOpenOptions mc QCryptoBlockOptionsBase mc QCryptoBlockOptionsLUKS mc QCryptoBlockOptionsQCow mc QCryptoCipherAlgorithm mc QCryptoCipherMode mc QCryptoHashAlgorithm mc QCryptoIVGenAlgorithm mc QCryptoSecretFormat mc QCryptoTLSCredsEndpoint mc QKeyCode mc QKeyCodeWrapper mc QMPCapability mc S390CrashReason mc SGXEPCSection mc SGXInfo mc SMPConfiguration mc TPMEmulatorOptions mc TPMEmulatorOptionsWrapper mc TPMInfo mc TPMPassthroughOptions mc TPMPassthroughOptionsWrapper mc X86CPUFeatureWordInfo mc X86CPURegister32 mc XBZRLECacheStats mc XDbgBlockGraph mc XDbgBlockGraphEdge mc XDbgBlockGraphNode mc XDbgBlockGraphNodeType mc ac_back mc ac_bookmarks mc ac_forward mc ac_home mc ac_refresh mc account_failed mc account_invalid mc add_client mc alt_r mc asl_compiler_id mc asl_compiler_rev mc avg_flush_latency_ns mc avg_rd_latency_ns mc avg_rd_queue_depth mc avg_wr_latency_ns mc avg_wr_queue_depth mc backing_file mc backing_file_depth mc block_resize mc block_set_io_throttle mc bps_max mc bps_max_length mc bps_rd mc bps_rd_max mc bps_rd_max_length mc bps_wr mc bps_wr_max mc bps_wr_max_length mc bracket_left mc bracket_right mc caps_lock mc class_info mc client_migrate_info mc cluster_alloc mc cluster_alloc_bytes mc cluster_alloc_space mc cluster_free mc compare_timeout mc cor_write mc cow_read mc cow_write mc ctrl_r mc d2_5 mc detect_zeroes mc device_add mc device_del mc empty_image_prepare mc expire_password mc expired_scan_cycle mc failed_flush_operations mc failed_rd_operations mc failed_unmap_operations mc failed_wr_operations mc flush_latency_histogram mc flush_operations mc flush_to_disk mc flush_to_os mc flush_total_time_ns mc grab_all mc grave_accent mc host_cdrom mc host_device mc idle_time_ns mc interval_length mc invalid_flush_operations mc invalid_rd_operations mc invalid_unmap_operations mc invalid_wr_operations mc io_range mc io_uring mc iops_max mc iops_max_length mc iops_rd mc iops_rd_max mc iops_rd_max_length mc iops_size mc iops_wr mc iops_wr_max mc iops_wr_max_length mc irq_pin mc known_hosts mc kp_0 mc kp_1 mc kp_2 mc kp_3 mc kp_4 mc kp_5 mc kp_6 mc kp_7 mc kp_8 mc kp_9 mc kp_add mc kp_comma mc kp_decimal mc kp_divide mc kp_enter mc kp_equals mc kp_multiply mc kp_subtract mc l1_grow_activate_table mc l1_grow_alloc_table mc l1_grow_write_table mc l1_shrink_free_l2_clusters mc l1_shrink_write_table mc l1_update mc l2_alloc_cow_read mc l2_alloc_write mc l2_load mc l2_update mc l2_update_compressed mc legacyESX mc max_flush_latency_ns mc max_queue_size mc max_rd_latency_ns mc max_wr_latency_ns mc mem_type_64 mc memory_range mc meta_l mc meta_r mc migrate_cancel mc min_flush_latency_ns mc min_rd_latency_ns mc min_wr_latency_ns mc monolithicFlat mc monolithicSparse mc netdev_add mc netdev_del mc new_state mc notify_dev mc num_lock mc oem_id mc oem_rev mc oem_table_id mc pci_bridge mc prefetchable_range mc primary_in mc pwritev_done mc pwritev_rmw_after_head mc pwritev_rmw_after_tail mc pwritev_rmw_head mc pwritev_rmw_tail mc pwritev_zero mc qdev_id mc qmp_capabilities mc rd_bytes mc rd_latency_histogram mc rd_merged mc rd_operations mc rd_total_time_ns mc read_aio mc read_backing_aio mc read_compressed mc refblock_alloc mc refblock_alloc_hookup mc refblock_alloc_switch_table mc refblock_alloc_write mc refblock_alloc_write_blocks mc refblock_alloc_write_table mc refblock_load mc refblock_update mc refblock_update_part mc reftable_grow mc reftable_load mc reftable_update mc sasl_username mc scroll_lock mc secondary_in mc secret_keyring mc set_link mc set_password mc shift_r mc streamOptimized mc system_powerdown mc system_reset mc system_wakeup mc timed_stats mc tray_open mc twoGbMaxExtentFlat mc twoGbMaxExtentSparse mc unmap_bytes mc unmap_merged mc unmap_operations mc unmap_total_time_ns mc vmstate_load mc vmstate_save mc vnet_hdr mc vnet_hdr_support mc wr_bytes mc wr_highest_offset mc wr_latency_histogram mc wr_merged mc wr_operations mc wr_total_time_ns mc write_aio mc write_compressed mc write_threshold mc x509_dname mc x86_64 uc ACPI_DEVICE_OST uc BALLOON_CHANGE uc BLOCK_EXPORT_DELETED uc BLOCK_IMAGE_CORRUPTED uc BLOCK_IO_ERROR uc BLOCK_JOB_CANCELLED uc BLOCK_JOB_COMPLETED uc BLOCK_JOB_ERROR uc BLOCK_JOB_PENDING uc BLOCK_JOB_READY uc BLOCK_WRITE_THRESHOLD uc COLO_EXIT uc CPU uc DEVICE_DELETED uc DEVICE_TRAY_MOVED uc DEVICE_UNPLUG_GUEST_ERROR uc DIMM uc DUMP_COMPLETED uc EAX uc EBP uc EBX uc ECX uc EDI uc EDX uc ESI uc ESP uc FAILOVER_NEGOTIATED uc GUEST_CRASHLOADED uc GUEST_PANICKED uc JOB_STATUS_CHANGE uc MEMORY_DEVICE_SIZE_CHANGE uc MEMORY_FAILURE uc MEM_UNPLUG_ERROR uc MIGRATION uc MIGRATION_PASS uc NIC_RX_FILTER_CHANGED uc POWERDOWN uc PR_MANAGER_STATUS_CHANGED uc QUORUM_FAILURE uc QUORUM_REPORT_BAD uc RDMA_GID_STATUS_CHANGED uc RESET uc RESUME uc RTC_CHANGE uc SHUTDOWN uc SPICE_CONNECTED uc SPICE_DISCONNECTED uc SPICE_INITIALIZED uc SPICE_MIGRATE_COMPLETED uc STOP uc SUSPEND uc SUSPEND_DISK uc UNPLUG_PRIMARY uc UUID uc VNC_CONNECTED uc VNC_DISCONNECTED uc VNC_INITIALIZED uc VSERPORT_CHANGE uc WAKEUP uc WATCHDOG
Quoted 13 lines · Andrea Bolognani
Quoted 1 line
Times how many naming conventions?
Yeah, I don't think requiring all possible permutations to be spelled out in the schema is the way to go. That's exactly why my proposal was to offer a way to inject the semantic information that the parser can't figure out itself.
Once you have a way to inform the generator that "VNCProps" is made of the two words "vnc" and "props", and that "vnc" is an acronym, then it can generate an identifier appropriate for the target language without having to spell out anywhere that such an identifier would be VNCProps for Go and VncProps for Rust.
Quoted 4 lines
By the way, while looking around I realized that we also have to take into account things like D-Bus: the QAPI type ChardevDBus, for example, would probably translate verbatim to Go but have to be changed to ChardevDbus for Rust. Fun :)
The hardest one of all is probably
QAuthZListPolicy
which must be split 'QAuthZ' + 'List' + 'Policy' - the trailing uppercase ruins all heuristics you can come up with, as there's no viable way to distinguish that scenario from 'ChardevDBus' which is 'Chardev' + 'DBus' not 'ChardevD' + 'Bus'
Quoted 3 lines
Revised proposal for the annotation:
ns:word-WORD-WoRD-123Word
Ugly, but we should only need this in the fairly niche scenarios, so not too pain ful to add a handful of these:
Essentially if have the schema use CamelCase with UPPERCASE acronyms, and declare two rules:
1. Split on every boundary from lower to upper
2. Acronym detection if there's a sequence of 3 uppercase letters, then split before the last uppercase.
then we'll do the right thing with the vast majority of cases:
ChardevSocket:
Rule 1: Chardev + Socket
Rule 2: Chardev + Socket
VNCProps:
Rule 1: VNCProps
Rule 2: VNC + Props
ChardevDBus
Rule 1: Chardev + DBus
Rule 2: Chardev + DBus
and fail on
QAuthZListPolicy
Rule 1: QAuth + ZList + Policy
Rule 2: QAuth + ZList + Policy
so only the last case needs ns:QAuthZ-List-Policy annotation, whcih
doesn't feel like a big burden
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 8 lines · Daniel P. Berrangé
Quoted 3 lines · Andrea Bolognani
Revised proposal for the annotation:
ns:word-WORD-WoRD-123Word
Ugly, but we should only need this in the fairly niche scenarios, so not too pain ful to add a handful of these:
Quoted 6 lines
Essentially if have the schema use CamelCase with UPPERCASE acronyms, and declare two rules:
1. Split on every boundary from lower to upper
2. Acronym detection if there's a sequence of 3 uppercase letters, then split before the last uppercase.
That should cover most of the type names, but we're still going to need to help the parser out when it comes to detecting acronyms in other contexts, such as all instances of the word "VNC" below:
{ 'enum': 'DisplayProtocol',
'data': [ 'vnc', ... ] }
{ 'command': 'query-vnc-servers', ... }
{ 'event': 'VNC_DISCONNECTED', ... }
Quoted 7 lines
QAuthZListPolicy
Rule 1: QAuth + ZList + Policy
Rule 2: QAuth + ZList + Policyso only the last case needs ns:QAuthZ-List-Policy annotation, whcih
doesn't feel like a big burden
Note that in my proposal the ns: part would be used exactly for cases like this one to separate the namespace part which, as you said in your other reply, needs to be preserved when generating C code but can be safely dropped when targeting a language that has actual namespace support. So the annotation would look like
Q:AuthZ-List-Policy
in this specific case. The ns: part would be optional, as a namespace is not embedded in most of the names.
It's also interesting to see how "AuthZ" is capitalized in various Go modules that implement the concept:
https://pkg.go.dev/search?limit=50&m=symbol&q=authz
Most use "Authz" rather than "AuthZ". If we get to the point where the only bit of weirdness is how we spell this specific word, I think I'll be able to live with it :)
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Quoted 14 lines · Daniel P. Berrangé
Quoted 13 lines · Andrea Bolognani
Quoted 1 line
Times how many naming conventions?
Yeah, I don't think requiring all possible permutations to be spelled out in the schema is the way to go. That's exactly why my proposal was to offer a way to inject the semantic information that the parser can't figure out itself.
Once you have a way to inform the generator that "VNCProps" is made of the two words "vnc" and "props", and that "vnc" is an acronym, then it can generate an identifier appropriate for the target language without having to spell out anywhere that such an identifier would be VNCProps for Go and VncProps for Rust.
Quoted 4 lines
Quoted 4 lines
By the way, while looking around I realized that we also have to take into account things like D-Bus: the QAPI type ChardevDBus, for example, would probably translate verbatim to Go but have to be changed to ChardevDbus for Rust. Fun :)
For what it's worth, I prefer Rust's style, because I hate downcasing tails of abbreviations a lot less than I hate WORDSRUNTOGETHER.
Quoted 23 lines
The hardest one of all is probably
QAuthZListPolicy
which must be split 'QAuthZ' + 'List' + 'Policy' - the trailing uppercase ruins all heuristics you can come up with, as there's no viable way to distinguish that scenario from 'ChardevDBus' which is 'Chardev' + 'DBus' not 'ChardevD' + 'Bus'
Quoted 3 lines
Revised proposal for the annotation:
ns:word-WORD-WoRD-123Word
Ugly, but we should only need this in the fairly niche scenarios, so not too pain ful to add a handful of these:
Essentially if have the schema use CamelCase with UPPERCASE acronyms, and declare two rules:
1. Split on every boundary from lower to upper
2. Acronym detection if there's a sequence of 3 uppercase letters, then split before the last uppercase.
Quoted 5 lines
then we'll do the right thing with the vast majority of cases:
ChardevSocket:
Rule 1: Chardev + Socket
Rule 2: Chardev + Socket
Rule 2 isn't used here.
Quoted 4 lines
VNCProps:
Rule 1: VNCProps
Rule 2: VNC + Props
Rule 2 is used here.
For Rust-style VncProps, rule 2 is again unused.
Quoted 2 lines
ChardevDBus
Note: "DBus" is not an (upper-case) acronym. It's a proper name grown from an abbreviation of "desktop bus".
Quoted 2 lines
Rule 1: Chardev + DBus
Rule 2: Chardev + DBus
Rule 2 isn't used here, either.
Quoted 9 lines
and fail on
QAuthZListPolicy
Rule 1: QAuth + ZList + Policy
Rule 2: QAuth + ZList + Policyso only the last case needs ns:QAuthZ-List-Policy annotation, whcih
doesn't feel like a big burden
I think there are two interesting sub-problems within the larger problem of mapping QAPI names to "nice" target language identifiers.
1. Splitting words run together (mostly camels, but also mistakes like logappend).
2. Adjust case in words for the target language
For 1., we can rely on word separators '-', '_' and change from lower to upper case. Cases where that doesn't suffice remain.
Adopting Go-style camels for QAPI will add to them, but something like your rule 2 above should mitigate.
What about this simple greedy algorithm:
funny_words = [QAuthZ, ...]
tail = name
words = []
while tail != "":
if tail starts with any of the words in funny_words:
next_word = that member of funny_words
else:
next_word = tail up to next separator or change from lower
to upper case
append next_word to tail
tail = remainder of tail with leading separator stripped
For 2., adjusting to all lower case (like "vnc", "props", "dbus"), all upper case (like "VNC", "PROPS", "DBUS"), and capitalized (like "Vnc", "Props", "Dbus") is trivial. Adjusting to capitalized except for funny words (like "VNC", "Props", "DBus") is not. Need a list of funny words, I guess.
Perhaps a single list of funnies would do for 1. and for 2.
Hi!
Sorry for taking some time to reply.
Quoted 9 lines · Andrea Bolognani
Quoted 2 lines · Victor Toso
Thanks for taking a look, let me know if you have questions, ideas or suggestions.
Full disclosure: I have only given the actual implementation a very cursory look so far, and I've focused on the generated Go API instead.
Overall things look pretty good.
Glad to hear.
Quoted 10 lines
One concern that I have is about naming struct members: things like SpiceInfo.MouseMode and most others are translated from the QAPI schema exactly the way you'd expect them, but for example ChardevCommon.Logappend doesn't look quite right. Of course there's no way to programmatically figure out what to capitalize, but maybe there's room for adding this kind of information in the form of additional annotations or something like that? Same for the various structs or members that have unexpectedly-capitalized "Tls" or "Vnc" in them.
Quoted 2 lines
To be clear, I don't think the above is a blocker - just something to be aware of, and think about.
There was a good discussion around this with Markus so I don't want to break it in another thread.
I'm happy that you have found those inconsistencies. I'll reply on the other thread about it but I don't mind working towards fixing it, either at code generator level or at QAPI level.
Quoted 24 lines
My biggest concern is about the interface offered for commands.
Based on the example you have in the README and how commands are defined, invoking (a simplified version of) the trace-event-get-state command would look like
cmd := Command{ Name: "trace-event-get-state", Arg: TraceEventGetStateCommand{ Name: "qemu_memalign", }, } qmp_input, _ := json.Marshal(&cmd) // qmp_input now contains // {"execute":"trace-event-get-state","arguments":{"name":"qemu_memalign"}} // do something with it
qmp_output:=
([]byte)({"return":{"name":"qemu_memalign","state":"disabled"}})
ret :=cmd.GetReturnType()
_ =json.Unmarshal(qmp_output, &ret)
// ret is a CommandResult instance whose Value member can be cast // to a TraceEventInfo struct
Quoted 2 lines
First of all, from an application's point of view there are way too many steps involved:
It can actually get worse. I've used a lot of nested struct to define a Base type for a given Type. In Go, If you try to initialize a Type that has a nested Struct, you'll need to use the nested struct Type as field name and this is too verbose.
See https://github.com/golang/go/issues/29438 (merged with:
https://github.com/golang/go/issues/12854)
The main reason that I kept it is because it maps very well with the over-the-wire protocol.
Quoted 8 lines
performing this operation should really be as
simple asret, _ :=
qmp.TraceEventGetState("qemu_memalign")// ret is a TraceEventInfo instanceThat's the end state we should be working towards.
Quoted 8 lines
Of course that assumes that the "qmp" object knows where the QMP socket is, knows how to talk the QMP protocol, transparently deals with serializing and deserializing data... Plus, in some case you might want to deal with the wire transfer yourself in an application-specific manner. So it makes sense to have the basic building blocks available and then build the more ergonomic SDK on top of that - with only the first part being in scope for this series.
Right. Indeed, I thought a bit about what I want to fit into the code generator that will reside in QEMU and what we might want to develop on top of that.
The goal for this series really is generating the data types that can be converted to/from QMP messages.
I completely agree with the message below: Type validation is important at this stage.
Quoted 7 lines
Even with that in mind, the current interface is IMO problematic because of its almost complete lack of type safety. Both Command.Arg and CommandResult.Value are of type Any and CommandBase.Name, which is used to drive the JSON unmarshal logic as well as ending up on the wire when executing a command, is just a plain string.
Quoted 7 lines
I think the low-level interface should look more like
cmd := TraceEventGetStateCommand{ Name: "qemu_memalign", } qmp_input, _ := json.Marshal(&cmd) // qmp_input looks the same as before
That isn't too hard to implement and I've started with this design at first. Each QAPI Command can implement a method Name() which returns the over-the-wire name for that Command.
I'm not yet sure if this is preferable over some other syntactic sugar function that might be generated (this series) or the next layer that will be on top of this.
But I agree with you that it should be improved before reaching actual Applications.
Quoted 10 lines
qmp_output:=
([]byte)({"return":{"name":"qemu_memalign","state":"disabled"}})
ret := TraceEventInfo{}
_ =json.Unmarshal(qmp_output, &ret)
// ret is a TraceEventInfo instanceThe advantages over the current implementation is that the compiler will prevent you from doing something silly like passing the wrong set of arguments to a commmand, and that the application has to explicitly spell out what kind of object it expects to get as output.
I think that, if we know all types that we can have at QAPI spec, the process of marshalling and unmarshalling should verify it. So, even if we don't change the expected interface as suggested, that work needs to be done. For some types, I've already did it, like for Unions and Alternate types.
Example: https://gitlab.com/victortoso/qapi-go/-/blob/main/pkg/qapi/unions.go#L28
This union type can have 4 values for the Any interface type. The code generator documents it to help user's out.
| type SocketAddressLegacy struct {
| // Base type for this struct
| SocketAddressLegacyBase
| // Value based on @type, possible types:
| // * InetSocketAddressWrapper
| // * UnixSocketAddressWrapper
| // * VsockSocketAddressWrapper
| // * StringWrapper
| Value Any
| }On the Marshal function, I used Sprintf as a way to fetch Value's type. There are other alternatives but to the cost of adding other deps.
| func (s SocketAddressLegacy) MarshalJSON() ([]byte, error) {
| base, err := json.Marshal(s.SocketAddressLegacyBase)
| if err != nil {
| return nil, err
| }
|
| typestr := fmt.Sprintf("%T", s.Value)
| typestr =
| typestr[strings.LastIndex(typestr, ".")+1:]...
| // "The branches need not cover all possible enum values"
| // This means that on Marshal, we can safely ignore empty values
| if typestr == "<nil>" {
| return []byte(base), nil
| }And then we have some Runtime checks to be sure to avoid the scenario mismatching Value's type.
| // Runtime check for supported value types
| if typestr != "StringWrapper" &&
| typestr != "InetSocketAddressWrapper" &&
| typestr != "UnixSocketAddressWrapper" &&
| typestr != "VsockSocketAddressWrapper" {
| return nil, errors.New(fmt.Sprintf("Type is not supported: %s", typestr))
| }
| value, err := json.Marshal(s.Value)
| if err != nil {
| return nil, err
| }With Alternate type, extra care was need on Unmarshal as we don't know the underlying type without looking at the message we received. That's the only reason of StrictDecode() helper function.
I'm just pointing out with above examples that I agree with you with Type safety. It is hard to infer everything at compile-time so we need some Runtime checks. Having some nicer APIs will definitely help and improve developer experience too.
Quoted 8 lines
I'm attaching an incomplete implementation that I used for playing around. It's obviously too simplistic, but hopefully it will help illustrate my point.
Dealing with errors and commands that don't have a return value might require us to have generic CommandResult wrapper after all, but we should really try as hard as we can to stick to type safe interfaces.
Agree. Many thanks again, for the review, suggestions and discussions.
Cheers,
Victor
Quoted 66 lines
Signature · 65 lines
Andrea Bolognani / Red Hat / Virtualization package main import ( "encoding/json" "fmt" ) type TraceEventGetStateCommand struct { Name string `json:"name"` } func (self *TraceEventGetStateCommand) MarshalJSON() ([]byte, error) { type Arguments TraceEventGetStateCommand return json.Marshal(&struct { Execute string `json:"execute"` Arguments *Arguments `json:"arguments"` }{ Execute: "trace-event-get-state", Arguments: (*Arguments)(self), }) } type TraceEventInfo struct { Name string `json:"name"` State string `json:"state"` } func (self *TraceEventInfo) UnmarshalJSON(data []byte) error { type Return TraceEventInfo ret := struct { Return Return `json:"return"` }{} err := json.Unmarshal(data, &ret) if err != nil { return err } self.Name = ret.Return.Name self.State = ret.Return.State return nil } func main() { var err error var qmp_input []byte var qmp_output []byte cmd := TraceEventGetStateCommand{ Name: "qemu_memalign", } if qmp_input, err = json.Marshal(&cmd); err != nil { panic(err) } fmt.Printf(" cmd: %v\n", cmd) fmt.Printf("-> qmp_input: %v\n", string(qmp_input)) qmp_output = ([]byte)(`{"return":{"name":"qemu_memalign","state":"disabled"}}`) ret := TraceEventInfo{} if err = json.Unmarshal(qmp_output, &ret); err != nil { panic(err) } fmt.Printf("<- qmp_output: %v\n", string(qmp_output)) fmt.Printf(" ret: %v\n", ret) }
Quoted 31 lines · Victor Toso
Quoted 22 lines · Andrea Bolognani
Based on the example you have in the README and how commands are defined, invoking (a simplified version of) the trace-event-get-state command would look like
cmd := Command{ Name: "trace-event-get-state", Arg: TraceEventGetStateCommand{ Name: "qemu_memalign", }, } qmp_input, _ := json.Marshal(&cmd) // qmp_input now contains // {"execute":"trace-event-get-state","arguments":{"name":"qemu_memalign"}} // do something with it
qmp_output:=
([]byte)({"return":{"name":"qemu_memalign","state":"disabled"}})
ret :=cmd.GetReturnType()
_ =json.Unmarshal(qmp_output, &ret)
// ret is a CommandResult instance whose Value member can be cast // to a TraceEventInfo structQuoted 2 lines
First of all, from an application's point of view there are way too many steps involved:
It can actually get worse. I've used a lot of nested struct to define a Base type for a given Type. In Go, If you try to initialize a Type that has a nested Struct, you'll need to use the nested struct Type as field name and this is too verbose.
Quoted 5 lines
See https://github.com/golang/go/issues/29438 (merged with:
https://github.com/golang/go/issues/12854)The main reason that I kept it is because it maps very well with the over-the-wire protocol.
Right, I had not realized how bad things really were :)
I've noticed the use of base types and while I didn't bring it up in my initial message because the other concerns seemed of much higher importance, I actually wondered whether we need to expose them to users of the Go SDK.
I think we should flatten things. That's what happens with the C generator already, for example in
struct InetSocketAddress {
/* Members inherited from InetSocketAddressBase: */
char *host;
char *port;
/* Own members: */
bool has_numeric;
bool numeric;
/* ... */
};This representation mirrors the wire protocol perfectly, so I see no reason not to adopt it. Am I missing something?
Quoted 17 lines
Quoted 8 lines
performing this operation should really be as
simple asret, _ :=
qmp.TraceEventGetState("qemu_memalign")// ret is a TraceEventInfo instanceThat's the end state we should be working towards.
Quoted 8 lines
Of course that assumes that the "qmp" object knows where the QMP socket is, knows how to talk the QMP protocol, transparently deals with serializing and deserializing data... Plus, in some case you might want to deal with the wire transfer yourself in an application-specific manner. So it makes sense to have the basic building blocks available and then build the more ergonomic SDK on top of that - with only the first part being in scope for this series.
Quoted 6 lines
Right. Indeed, I thought a bit about what I want to fit into the code generator that will reside in QEMU and what we might want to develop on top of that.
The goal for this series really is generating the data types that can be converted to/from QMP messages.
That's perfectly fine, and in fact I believe that splitting the whole endeavor into three parts - QMP protocol implementation, QAPI types serialization/deserialization, and a high-level SDK that gives easy access to the previous two - is the best approach.
Quoted 75 lines
Quoted 10 lines
qmp_output:=
([]byte)({"return":{"name":"qemu_memalign","state":"disabled"}})
ret := TraceEventInfo{}
_ =json.Unmarshal(qmp_output, &ret)
// ret is a TraceEventInfo instanceThe advantages over the current implementation is that the compiler will prevent you from doing something silly like passing the wrong set of arguments to a commmand, and that the application has to explicitly spell out what kind of object it expects to get as output.
I think that, if we know all types that we can have at QAPI spec, the process of marshalling and unmarshalling should verify it. So, even if we don't change the expected interface as suggested, that work needs to be done. For some types, I've already did it, like for Unions and Alternate types.
Example: https://gitlab.com/victortoso/qapi-go/-/blob/main/pkg/qapi/unions.go#L28
This union type can have 4 values for the Any interface type. The code generator documents it to help user's out.
| type SocketAddressLegacy struct { | // Base type for this struct | SocketAddressLegacyBase | // Value based on @type, possible types: | // * InetSocketAddressWrapper | // * UnixSocketAddressWrapper | // * VsockSocketAddressWrapper | // * StringWrapper | Value Any | }On the Marshal function, I used Sprintf as a way to fetch Value's type. There are other alternatives but to the cost of adding other deps.
| func (s SocketAddressLegacy) MarshalJSON() ([]byte, error) { | base, err := json.Marshal(s.SocketAddressLegacyBase) | if err != nil { | return nil, err | } | | typestr := fmt.Sprintf("%T", s.Value) | typestr = | typestr[strings.LastIndex(typestr, ".")+1:]...
| // "The branches need not cover all possible enum values" | // This means that on Marshal, we can safely ignore empty values | if typestr == "<nil>" { | return []byte(base), nil | }And then we have some Runtime checks to be sure to avoid the scenario mismatching Value's type.
| // Runtime check for supported value types | if typestr != "StringWrapper" && | typestr != "InetSocketAddressWrapper" && | typestr != "UnixSocketAddressWrapper" && | typestr != "VsockSocketAddressWrapper" { | return nil, errors.New(fmt.Sprintf("Type is not supported: %s", typestr)) | } | value, err := json.Marshal(s.Value) | if err != nil { | return nil, err | }With Alternate type, extra care was need on Unmarshal as we don't know the underlying type without looking at the message we received. That's the only reason of
StrictDecode()helper function.
Quoted 4 lines
I'm just pointing out with above examples that I agree with you with Type safety. It is hard to infer everything at compile-time so we need some Runtime checks. Having some nicer APIs will definitely help and improve developer experience too.
I agree that in some cases build time validation is simply not possible, but this doesn't seem to be one of those cases.
For unmarshaling? Sure, especially if we keep in mind the forward compatibility scenarios that Dan has raised elsewhere in the thread, and that to be completely honest I haven't gotten around to fully digesting yet.
But when it comes to marshaling we know ahead of time that the value is going to be one of a handful of types, and we should get the compiler to spot violations of this assumption for us. Using "Any" prevents that.
If you look at libvirt-go-xml-module, alternates are handled with things like
type DomainInterfaceSource struct {
Ethernet *DomainInterfaceSourceEthernet `xml:"-"`
Server *DomainInterfaceSourceServer `xml:"-"`
// ...
}
type DomainInterfaceSourceEthernet struct {
IP []DomainInterfaceIP `xml:"ip"`
Route []DomainInterfaceRoute `xml:"route"`
}
type DomainInterfaceSourceServer struct {
Address string `xml:"address,attr,omitempty"`
Port uint `xml:"port,attr,omitempty"`
Local *DomainInterfaceSourceLocal `xml:"local"`
}and (un)marshaling is done like
func (a *DomainInterfaceSource) MarshalXML(e *xml.Encoder, start
xml.StartElement) error {
if a.User != nil {
return nil
} else if a.Ethernet != nil {
if len(a.Ethernet.IP) > 0 && len(a.Ethernet.Route) > 0 {
return e.EncodeElement(a.Ethernet, start)
}
return nil
} else if // ...}
func (a *DomainInterfaceSource) UnmarshalXML(d *xml.Decoder, start
xml.StartElement) error {
if a.User != nil {
return d.DecodeElement(a.User, &start)
} else if a.Ethernet != nil {
return d.DecodeElement(a.Ethernet, &start)
} else if // ...}
When using these data structures, you'll then do something like
iface := DomainInterface{
Source: &DomainInterfaceSource{
Network: &DomainInterfaceNetwork{
Network: "default",
},
},
}instead of
iface := DomainInterface{
Source: DomainInterfaceSource{
Type: "network",
Value: DomainInterfaceNetwork{
Network: "default",
},
},
}which is more compact and, crucially, allows the compiler to catch many typing issues ahead of time. I think this approach would work really well for QAPI too.
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Quoted 6 lines · Andrea Bolognani
Quoted 5 lines · Victor Toso
Quoted 4 lines · Andrea Bolognani
Based on the example you have in the README and how commands are defined, invoking (a simplified version of) the trace-event-get-state command would look like
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
cmd := Command{ Name: "trace-event-get-state", Arg: TraceEventGetStateCommand{ Name: "qemu_memalign", }, }
Note there is clear redundancy here between 'Name' and the struct type. IMHO the better approach would be
cmd := TraceEventGetStateCommand{
Name: 'qemu_memalign'
}and have 'Command' simply as an interface that TraceEventGetStateCommand implements. I don't think the interface would need more than a Marshal and Demarshal method, which serialize the 'Arg' and then add the Name field explicitly.
Quoted 46 lines
Quoted 20 lines
Quoted 12 lines
qmp_input, _ :=json.Marshal(&cmd)
//qmp_inputnow contains
// {"execute":"trace-event-get-state","arguments":{"name":"qemu_memalign"}}
// do something with it
qmp_output:=
([]byte)({"return":{"name":"qemu_memalign","state":"disabled"}})
ret :=cmd.GetReturnType()
_ =json.Unmarshal(qmp_output, &ret)
// ret is a CommandResult instance whose Value member can be cast // to a TraceEventInfo structQuoted 2 lines
First of all, from an application's point of view there are way too many steps involved:
It can actually get worse. I've used a lot of nested struct to define a Base type for a given Type. In Go, If you try to initialize a Type that has a nested Struct, you'll need to use the nested struct Type as field name and this is too verbose.
Quoted 5 lines
See https://github.com/golang/go/issues/29438 (merged with:
https://github.com/golang/go/issues/12854)The main reason that I kept it is because it maps very well with the over-the-wire protocol.
Right, I had not realized how bad things really were :)
I've noticed the use of base types and while I didn't bring it up in my initial message because the other concerns seemed of much higher importance, I actually wondered whether we need to expose them to users of the Go SDK.
I think we should flatten things. That's what happens with the C generator already, for example in
struct InetSocketAddress { /* Members inherited from InetSocketAddressBase: */ char *host; char *port; /* Own members: */ bool has_numeric; bool numeric; /* ... */ };
Quoted 2 lines
This representation mirrors the wire protocol perfectly, so I see no reason not to adopt it. Am I missing something?
The main reason not to flatten is if you have scenarios where it is useful to work against the base type directly. I'm not sure that we have such a need though.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 37 lines · Victor Toso
Hi,
Happy 1st April. Not a joke :) /* ugh, took me too long to send */
This series is about adding a generator in
scripts/qapito produce Go data structures that can be used to communicate with QEMU over QMP.* Why Go?
There are quite a few Go projects that interact with QEMU over QMP and they endup using a mix of different libraries with their own code.
** Which projects?
The ones I've found so far:
- podman machine
https://github.com/containers/podman/tree/main/pkg/machine/qemu- kata-containers (govmm)
https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm- lxd
https://github.com/lxc/lxd/tree/master/lxd/instance/drivers- kubevirt (plain json strings)
https://github.com/kubevirt/kubevirt(let me know if you know others)
* But Why?
Quoted 8 lines
I'm particularly interested in 3 out of 4 of the projects above and only Kubevirt uses libvirt to handle QEMU. That means that every QEMU releases where a QMP command, event or other data struct is added, removed or changed, those projects need to check what changed in QEMU and then address those changes in their projects, if needed.
The idea behind generating Go data structures is that we can keep a Go module which can have releases that follow QEMU releases.
We need to look at "following the QEMU releases" a bit more closely.
Merging your patches gives us the capability to generate a Go interface to HEAD's version of QMP.
The obvious way for an out-of-tree Go program to use this generated Go interface is to build with a specific version of it. It can then talk QMP to any compatible QEMU version.
Compatibility with older QEMUs is not assured: stuff added since is present on the Go QMP client end, but not on the QEMU QMP server end.
Compatibility with newer QEMUs is subject to our deprecation policy:
In general features are intended to be supported indefinitely once introduced into QEMU. In the event that a feature needs to be removed, it will be listed in this section. The feature will remain functional for the release in which it was deprecated and one further release. After these two releases, the feature is liable to be removed.
So, if you stay away from deprecated stuff, you're good for two more releases at least.
Does this work for the projects you have in mind?
Aside: graceful degradation in case of incompatibility seems desirable.
Quoted 3 lines
The project that uses this Go module, only need to bump the module version and it shall receive all the changes in their own vendored code base.
Ideally, incompatible changes that affect the Go program show up as compile errors. Do they?
Quoted 17 lines
* Status
There are a few rough edges to work on but this is usable. The major thing I forgot to add is handling Error from Commands. It'll be the first thing I'll work on next week.
If you want to start using this Today you can fetch it in at
https://gitlab.com/victortoso/qapi-go/
There are quite a few tests that I took from the examples in the qapi schema. Coverage using go's cover tool is giving
28.6% of statementsI've uploaded the a static generated godoc output of the above Go module here:
Quoted 6 lines
https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/
* License
While the generator (
golang.pyin this series) is GPL v2, the
I'd make it v2+, just to express my displeasure with the decision to make the initial QAPI generator v2 only for no good reason at all.
Quoted 3 lines
generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.
Meh. Can't be helped, I guess.
Quoted 11 lines
* Disclaimer to reviewers
This is my first serious python project so there'll be lots of suggetions that I'll be happy to take and learn from.
Thanks for taking a look, let me know if you have questions, ideas or suggestions.
Cheers,
Victor
Hi,
Thanks for the quick review Markus. Sorry for taking quite a bit of time to get back to you.
Quoted 45 lines · Markus Armbruster
Quoted 43 lines · Victor Toso
Hi,
Happy 1st April. Not a joke :) /* ugh, took me too long to send */
This series is about adding a generator in
scripts/qapito produce Go data structures that can be used to communicate with QEMU over QMP.* Why Go?
There are quite a few Go projects that interact with QEMU over QMP and they endup using a mix of different libraries with their own code.
** Which projects?
The ones I've found so far:
- podman machine
https://github.com/containers/podman/tree/main/pkg/machine/qemu- kata-containers (govmm)
https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm- lxd
https://github.com/lxc/lxd/tree/master/lxd/instance/drivers- kubevirt (plain json strings)
https://github.com/kubevirt/kubevirt(let me know if you know others)
* But Why?
I'm particularly interested in 3 out of 4 of the projects above and only Kubevirt uses libvirt to handle QEMU. That means that every QEMU releases where a QMP command, event or other data struct is added, removed or changed, those projects need to check what changed in QEMU and then address those changes in their projects, if needed.
Quoted 8 lines
Quoted 2 lines
The idea behind generating Go data structures is that we can keep a Go module which can have releases that follow QEMU releases.
We need to look at "following the QEMU releases" a bit more closely.
Merging your patches gives us the capability to generate a Go interface to HEAD's version of QMP.
Right, just to put it up here, it should be expected that the qapi-go project is to have releases that match QEMU's release.
Quoted 19 lines
The obvious way for an out-of-tree Go program to use this generated Go interface is to build with a specific version of it. It can then talk QMP to any compatible QEMU version.
Compatibility with older QEMUs is not assured: stuff added since is present on the Go QMP client end, but not on the QEMU QMP server end.
Compatibility with newer QEMUs is subject to our deprecation policy:
In general features are intended to be supported indefinitely once introduced into QEMU. In the event that a feature needs to be removed, it will be listed in this section. The feature will remain functional for the release in which it was deprecated and one further release. After these two releases, the feature is liable to be removed.
Quoted 4 lines
So, if you stay away from deprecated stuff, you're good for two more releases at least.
Does this work for the projects you have in mind?
It depends on how the project will be using qapi-go, so I can't say for them all.
There are projects that will be targeting specific QEMU version (e.g: Kubevirt, Kata containers) and for those, I think they don't mind only bumping qapi-go when they plan to change the target QEMU version or perhaps to keep separated binary versions for a limited amount of time (just my guess).
Some other projects like Podman, will likely be talking with the running version of QEMU they have in that host. The possibilities are quite broad here.
Quoted 2 lines
Aside: graceful degradation in case of incompatibility seems desirable.
I agree. I haven't thought much on how to handle those scenarios yet and suggestions are more than welcomed.
I know that, those projects are already hardcoding commands and expected return types by hand. My first goal is to provide well defined types, QAPI/QMP compliant, with easy to reach documentation as provided by QAPI docs.
I expect that, step by step, we can improve things all around but I don't expect it to be done all at once.
Quoted 6 lines
Quoted 3 lines
The project that uses this Go module, only need to bump the module version and it shall receive all the changes in their own vendored code base.
Ideally, incompatible changes that affect the Go program show up as compile errors. Do they?
It depends. A field/type that they were using but is removed, for sure a compile-time error.
What about a new mandatory field? If you run the unit tests in qapi-go, there will be failures but it wouldn't be compile time error. If user doesn't define a mandatory field in a Go struct, the default value is used for the over-the-wire message.
Perhaps some tooling can be developed to help users check that something they are using has changed. I'll look into it.
Quoted 28 lines
Quoted 17 lines
* Status
There are a few rough edges to work on but this is usable. The major thing I forgot to add is handling Error from Commands. It'll be the first thing I'll work on next week.
If you want to start using this Today you can fetch it in at
https://gitlab.com/victortoso/qapi-go/
There are quite a few tests that I took from the examples in the qapi schema. Coverage using go's cover tool is giving
28.6% of statementsI've uploaded the a static generated godoc output of the above Go module here:
Quoted 6 lines
https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/
* License
While the generator (
golang.pyin this series) is GPL v2, theI'd make it v2+, just to express my displeasure with the decision to make the initial QAPI generator v2 only for no good reason at all.
Quoted 5 lines
Quoted 3 lines
generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.
Meh. Can't be helped, I guess.
:)
Quoted 11 lines
Quoted 11 lines
* Disclaimer to reviewers
This is my first serious python project so there'll be lots of suggetions that I'll be happy to take and learn from.
Thanks for taking a look, let me know if you have questions, ideas or suggestions.
Cheers,
Victor
Thanks again,
Victor
Quoted 69 lines · Markus Armbruster
Quoted 37 lines · Victor Toso
Hi,
Happy 1st April. Not a joke :) /* ugh, took me too long to send */
This series is about adding a generator in
scripts/qapito produce Go data structures that can be used to communicate with QEMU over QMP.* Why Go?
There are quite a few Go projects that interact with QEMU over QMP and they endup using a mix of different libraries with their own code.
** Which projects?
The ones I've found so far:
- podman machine
https://github.com/containers/podman/tree/main/pkg/machine/qemu- kata-containers (govmm)
https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm- lxd
https://github.com/lxc/lxd/tree/master/lxd/instance/drivers- kubevirt (plain json strings)
https://github.com/kubevirt/kubevirt(let me know if you know others)
* But Why?
Quoted 8 lines
I'm particularly interested in 3 out of 4 of the projects above and only Kubevirt uses libvirt to handle QEMU. That means that every QEMU releases where a QMP command, event or other data struct is added, removed or changed, those projects need to check what changed in QEMU and then address those changes in their projects, if needed.
The idea behind generating Go data structures is that we can keep a Go module which can have releases that follow QEMU releases.
We need to look at "following the QEMU releases" a bit more closely.
Merging your patches gives us the capability to generate a Go interface to HEAD's version of QMP.
The obvious way for an out-of-tree Go program to use this generated Go interface is to build with a specific version of it. It can then talk QMP to any compatible QEMU version.
Compatibility with older QEMUs is not assured: stuff added since is present on the Go QMP client end, but not on the QEMU QMP server end.
Compatibility with newer QEMUs is subject to our deprecation policy:
In general features are intended to be supported indefinitely once introduced into QEMU. In the event that a feature needs to be removed, it will be listed in this section. The feature will remain functional for the release in which it was deprecated and one further release. After these two releases, the feature is liable to be removed.
Quoted 4 lines
So, if you stay away from deprecated stuff, you're good for two more releases at least.
Does this work for the projects you have in mind?
It might work for some projects, but in the general case I find it pretty unappealing as a restriction. Mixing and matching new QEMU with old libvirt, or vica-verca has been an incredibly common thing todo when both developing and perhaps more importantly debugging problems. For example I have one libvirt build and I use it against any QEMU from Fedora / any RHEL-8.x update, which spans a great many QEMU releases.
I like the idea of auto-generating clients from the QAPI schema, and would like it if we were able to use this kind of approach on the libvirt side, but for that we need to be more flexible in version matching.
Our current approach to deprecation features and subsequently removing them from the QAPI schema works fine when the QAPI schema is only used internally by QEMU, not when we we expand usage of QAPI to external applications.
I think we need to figure out a way to make the QAPI schema itself be append only, while still allowing QEMU to deprecation & remove features.
For a minimum viable use case, this doesn't feel all that difficult, as conceptually instead of deleting the field from QAPI, we just need to annotate it to say when it was deleted from the QEMU side. The QAPI generator for internal QEMU usage, can omit any fields annotated as deleted in QAPI schema. The QAPI generator for external app usage, can (optionally) be told to include deleted fields ranging back to a given version number. So apps can chooses what degree of compat they wish to retain.
Apps that wish to have version compat, would of course need to write their code to be aware of which fields they need to seend for which QEMU version.
Quoted 20 lines
Quoted 20 lines
* Status
There are a few rough edges to work on but this is usable. The major thing I forgot to add is handling Error from Commands. It'll be the first thing I'll work on next week.
If you want to start using this Today you can fetch it in at
https://gitlab.com/victortoso/qapi-go/
There are quite a few tests that I took from the examples in the qapi schema. Coverage using go's cover tool is giving
28.6% of statementsI've uploaded the a static generated godoc output of the above Go module here:
https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/
Quoted 6 lines
Quoted 3 lines
* License
While the generator (
golang.pyin this series) is GPL v2, theI'd make it v2+, just to express my displeasure with the decision to make the initial QAPI generator v2 only for no good reason at all.
Our policy is that all new code should be v2+ anyway, unless it was clearly derived from some pre-existing v2-only code. Upto Victor to say whether the golang.py is considered clean, or was copy+paste in any parts from existin v2-only code
Quoted 5 lines
Quoted 3 lines
generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.
Meh. Can't be helped, I guess.
This does make me wonder though whether the license of the QAPI input files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is the Go code to be considered a derived work of the QAPI JSON files. I'm not finding a clearly articulated POV on this question so far.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hi,
Quoted 11 lines · Daniel P. Berrangé
Quoted 6 lines
Quoted 3 lines
* License
While the generator (
golang.pyin this series) is GPL v2, theI'd make it v2+, just to express my displeasure with the decision to make the initial QAPI generator v2 only for no good reason at all.
Our policy is that all new code should be v2+ anyway, unless it was clearly derived from some pre-existing v2-only code. Upto Victor to say whether the
golang.pyis considered clean, or was copy+paste in any parts from existin v2-only code
Should be fine to consider it v2+, the generator.
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.
Meh. Can't be helped, I guess.
This does make me wonder though whether the license of the QAPI input files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is the Go code to be considered a derived work of the QAPI JSON files.
GPL does not enforce that the generated code to be GPL [0] unless the generator copies GPL code to the output. My only concern has been the fact that I am copying the documentation of QAPI specification to the Go package in order to have data structures, commands, etc. with the information provided by the specification.
[0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput
Quoted 2 lines
I'm not finding a clearly articulated POV on this question so far.
I don't find it trivial either but I've accepted that the Go data structures are fine based on [0] and the documentation can be split from the Go module (sadly!) if someone finds it to be a legal issue.
Cheers,
Victor
Quoted 37 lines · Victor Toso
Hi,
Quoted 11 lines · Daniel P. Berrangé
Quoted 6 lines
Quoted 3 lines
* License
While the generator (
golang.pyin this series) is GPL v2, theI'd make it v2+, just to express my displeasure with the decision to make the initial QAPI generator v2 only for no good reason at all.
Our policy is that all new code should be v2+ anyway, unless it was clearly derived from some pre-existing v2-only code. Upto Victor to say whether the
golang.pyis considered clean, or was copy+paste in any parts from existin v2-only codeShould be fine to consider it v2+, the generator.
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.
Meh. Can't be helped, I guess.
This does make me wonder though whether the license of the QAPI input files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is the Go code to be considered a derived work of the QAPI JSON files.
GPL does not enforce that the generated code to be GPL [0] unless the generator copies GPL code to the output. My only concern has been the fact that I am copying the documentation of QAPI specification to the Go package in order to have data structures, commands, etc. with the information provided by the specification.
[0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput
Quoted 7 lines
Quoted 2 lines
I'm not finding a clearly articulated POV on this question so far.
I don't find it trivial either but I've accepted that the Go data structures are fine based on [0] and the documentation can be split from the Go module (sadly!) if someone finds it to be a legal issue.
Ah well that link above is actually reasonably clear:
"More generally, when a program translates its input into
some other form, the copyright status of the output inherits that of the input it was generated from. "
In our case the input is the QAPI JSON, and we're translating that into Golang. That could be read as meaning our Golang code has to be GPLv2+ licensed just as with the QAPI, not merely the docs.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 48 lines · Daniel P. Berrangé
Quoted 37 lines · Victor Toso
Hi,
Quoted 11 lines · Daniel P. Berrangé
Quoted 6 lines
Quoted 3 lines
* License
While the generator (
golang.pyin this series) is GPL v2, theI'd make it v2+, just to express my displeasure with the decision to make the initial QAPI generator v2 only for no good reason at all.
Our policy is that all new code should be v2+ anyway, unless it was clearly derived from some pre-existing v2-only code. Upto Victor to say whether the
golang.pyis considered clean, or was copy+paste in any parts from existin v2-only codeShould be fine to consider it v2+, the generator.
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.
Meh. Can't be helped, I guess.
This does make me wonder though whether the license of the QAPI input files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is the Go code to be considered a derived work of the QAPI JSON files.
GPL does not enforce that the generated code to be GPL [0] unless the generator copies GPL code to the output. My only concern has been the fact that I am copying the documentation of QAPI specification to the Go package in order to have data structures, commands, etc. with the information provided by the specification.
[0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput
Quoted 7 lines
Quoted 2 lines
I'm not finding a clearly articulated POV on this question so far.
I don't find it trivial either but I've accepted that the Go data structures are fine based on [0] and the documentation can be split from the Go module (sadly!) if someone finds it to be a legal issue.
Ah well that link above is actually reasonably clear:
Quoted 7 lines
"More generally, when a program translates its input into
some other form, the copyright status of the output inherits that of the input it was generated from. "In our case the input is the QAPI JSON, and we're translating that into Golang. That could be read as meaning our Golang code has to be GPLv2+ licensed just as with the QAPI, not merely the docs.
Oh but I'm forgetting that QAPI JSON can be said to be our public API interface, and so while the docs text would be GPLv2+, we can claim fair use for the interface definition in the generator and pick an arbitrary output license.
We could likely deal with the docs problem by not copying the docs directly into the generated code, instead link to the published docs on qemu.org. This would require us to improve our current docs generated anchor generation. ie currently docs link for say the struct 'NumaNodeOptions' potentially changes every time we generate it
https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#qapidoc-2416
we would need to that be something stable, ie
https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions
Then the generated Go can just do
// See QAPI docs at https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions
thus avoiding any copyright complication
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hi,
Quoted 36 lines · Daniel P. Berrangé
Quoted 35 lines · Daniel P. Berrangé
Quoted 20 lines · Victor Toso
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.
Meh. Can't be helped, I guess.
This does make me wonder though whether the license of the QAPI input files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is the Go code to be considered a derived work of the QAPI JSON files.
GPL does not enforce that the generated code to be GPL [0] unless the generator copies GPL code to the output. My only concern has been the fact that I am copying the documentation of QAPI specification to the Go package in order to have data structures, commands, etc. with the information provided by the specification.
[0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput
Quoted 7 lines
Quoted 2 lines
I'm not finding a clearly articulated POV on this question so far.
I don't find it trivial either but I've accepted that the Go data structures are fine based on [0] and the documentation can be split from the Go module (sadly!) if someone finds it to be a legal issue.
Ah well that link above is actually reasonably clear:
"More generally, when a program translates its input into
some other form, the copyright status of the output inherits that of the input it was generated from. "
Quoted 8 lines
Quoted 3 lines
In our case the input is the QAPI JSON, and we're translating that into Golang. That could be read as meaning our Golang code has to be GPLv2+ licensed just as with the QAPI, not merely the docs.
Oh but I'm forgetting that QAPI JSON can be said to be our public API interface, and so while the docs text would be GPLv2+, we can claim fair use for the interface definition in the generator and pick an arbitrary output license.
Still, it explicit says in the section "In what cases is the output of a GPL program covered by the GPL too?" is " Only when the program copies part of itself into the output".
https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#WhatCaseIsOutputGPL
So, to my understating, even if we are consuming a GPLv2+ spec with a GPLv2+ generator, the output does not need to be GPLv2+ too, unless we are *copying* parts of the input/generator into the output - which is the case for the documentation only.
I'll raise this again with the my company's legal team to be sure.
Quoted 11 lines
We could likely deal with the docs problem by not copying the docs directly into the generated code, instead link to the published docs on qemu.org. This would require us to improve our current docs generated anchor generation. ie currently docs link for say the struct 'NumaNodeOptions' potentially changes every time we generate it
https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#qapidoc-2416
we would need to that be something stable, ie
Quoted 7 lines
https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions
Then the generated Go can just do
// See QAPI docs at https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptionsthus avoiding any copyright complication
Yes, but it would be quite sad solution. Documentation in Go is bounded to the module source code and we would be making people jump thorough links here.
I mean, if that's what we need to do, okay.
I'll keep thinking about alternatives.
Cheers,
Victor
Quoted 79 lines · Victor Toso
Hi,
Quoted 36 lines · Daniel P. Berrangé
Quoted 35 lines · Daniel P. Berrangé
Quoted 20 lines · Victor Toso
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.
Meh. Can't be helped, I guess.
This does make me wonder though whether the license of the QAPI input files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is the Go code to be considered a derived work of the QAPI JSON files.
GPL does not enforce that the generated code to be GPL [0] unless the generator copies GPL code to the output. My only concern has been the fact that I am copying the documentation of QAPI specification to the Go package in order to have data structures, commands, etc. with the information provided by the specification.
[0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput
Quoted 7 lines
Quoted 2 lines
I'm not finding a clearly articulated POV on this question so far.
I don't find it trivial either but I've accepted that the Go data structures are fine based on [0] and the documentation can be split from the Go module (sadly!) if someone finds it to be a legal issue.
Ah well that link above is actually reasonably clear:
"More generally, when a program translates its input into
some other form, the copyright status of the output inherits that of the input it was generated from. "Quoted 8 lines
Quoted 3 lines
In our case the input is the QAPI JSON, and we're translating that into Golang. That could be read as meaning our Golang code has to be GPLv2+ licensed just as with the QAPI, not merely the docs.
Oh but I'm forgetting that QAPI JSON can be said to be our public API interface, and so while the docs text would be GPLv2+, we can claim fair use for the interface definition in the generator and pick an arbitrary output license.
Still, it explicit says in the section "In what cases is the output of a GPL program covered by the GPL too?" is " Only when the program copies part of itself into the output".
https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#WhatCaseIsOutputGPL
So, to my understating, even if we are consuming a GPLv2+ spec with a GPLv2+ generator, the output does not need to be GPLv2+ too, unless we are *copying* parts of the input/generator into the output - which is the case for the documentation only.
I'll raise this again with the my company's legal team to be sure.
Quoted 17 lines
We could likely deal with the docs problem by not copying the docs directly into the generated code, instead link to the published docs on qemu.org. This would require us to improve our current docs generated anchor generation. ie currently docs link for say the struct 'NumaNodeOptions' potentially changes every time we generate it
https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#qapidoc-2416
we would need to that be something stable, ie
https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions
Then the generated Go can just do
// See QAPI docs at https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions
Quoted 7 lines
Quoted 1 line
thus avoiding any copyright complication
Yes, but it would be quite sad solution. Documentation in Go is bounded to the module source code and we would be making people jump thorough links here.
I mean, if that's what we need to do, okay.
It isn't the end of the world IMHO, as people are typically browsing docs from the online docs site, so full details are only a click away.
This is what I did in libvirt Go APIs for example
https://pkg.go.dev/libvirt.org/go/libvirt#Domain.SnapshotLookupByName
I feel the biggest impact for developers is actually the quality of the docs that exist. Time invested in better QAPI docs will have more impact on developers, than trying to eliminate the need to follow one extra hyperlink.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 73 lines · Daniel P. Berrangé
Quoted 72 lines · Markus Armbruster
Quoted 37 lines · Victor Toso
Hi,
Happy 1st April. Not a joke :) /* ugh, took me too long to send */
This series is about adding a generator in
scripts/qapito produce Go data structures that can be used to communicate with QEMU over QMP.* Why Go?
There are quite a few Go projects that interact with QEMU over QMP and they endup using a mix of different libraries with their own code.
** Which projects?
The ones I've found so far:
- podman machine
https://github.com/containers/podman/tree/main/pkg/machine/qemu- kata-containers (govmm)
https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm- lxd
https://github.com/lxc/lxd/tree/master/lxd/instance/drivers- kubevirt (plain json strings)
https://github.com/kubevirt/kubevirt(let me know if you know others)
* But Why?
Quoted 8 lines
I'm particularly interested in 3 out of 4 of the projects above and only Kubevirt uses libvirt to handle QEMU. That means that every QEMU releases where a QMP command, event or other data struct is added, removed or changed, those projects need to check what changed in QEMU and then address those changes in their projects, if needed.
The idea behind generating Go data structures is that we can keep a Go module which can have releases that follow QEMU releases.
We need to look at "following the QEMU releases" a bit more closely.
Merging your patches gives us the capability to generate a Go interface to HEAD's version of QMP.
The obvious way for an out-of-tree Go program to use this generated Go interface is to build with a specific version of it. It can then talk QMP to any compatible QEMU version.
Compatibility with older QEMUs is not assured: stuff added since is present on the Go QMP client end, but not on the QEMU QMP server end.
Compatibility with newer QEMUs is subject to our deprecation policy:
In general features are intended to be supported indefinitely once introduced into QEMU. In the event that a feature needs to be removed, it will be listed in this section. The feature will remain functional for the release in which it was deprecated and one further release. After these two releases, the feature is liable to be removed.
So, if you stay away from deprecated stuff, you're good for two more releases at least.
Quoted 8 lines
Quoted 1 line
Does this work for the projects you have in mind?
It might work for some projects, but in the general case I find it pretty unappealing as a restriction. Mixing and matching new QEMU with old libvirt, or vica-verca has been an incredibly common thing todo when both developing and perhaps more importantly debugging problems. For example I have one libvirt build and I use it against any QEMU from Fedora / any RHEL-8.x update, which spans a great many QEMU releases.
I'd like to propose that for compatibility with a wide range of QEMU versions, you use or reinvent libvirt.
Quoted 11 lines
I like the idea of auto-generating clients from the QAPI schema, and would like it if we were able to use this kind of approach on the libvirt side, but for that we need to be more flexible in version matching.
Our current approach to deprecation features and subsequently removing them from the QAPI schema works fine when the QAPI schema is only used internally by QEMU, not when we we expand usage of QAPI to external applications.
I think we need to figure out a way to make the QAPI schema itself be append only, while still allowing QEMU to deprecation & remove features.
This is going to get complicated fast.
Quoted 8 lines
For a minimum viable use case, this doesn't feel all that difficult, as conceptually instead of deleting the field from QAPI, we just need to annotate it to say when it was deleted from the QEMU side. The QAPI generator for internal QEMU usage, can omit any fields annotated as deleted in QAPI schema. The QAPI generator for external app usage, can (optionally) be told to include deleted fields ranging back to a given version number. So apps can chooses what degree of compat they wish to retain.
Consider this evolution of command block_resize
* Initially, it has a mandatory argument @device[*].
* An alternative way to specify the command's object emerges: new argument @node-name. Both old @device and new @node-name become optional, and exactly one of them must be specified. This is commit 3b1dbd11a6 "qmp: Allow block_resize to manipulate bs graph nodes."
* At some future date, the old way gets deprecated: argument @device acquires feature @deprecated.
* Still later, the old way gets removed: @device is deleted, and @node-name becomes mandatory.
What is the proper version-spanning interface?
I figure it's both arguments optional, must specify the right one for the version of QEMU actually in use. This spans versions, but it fails to abstract from them.
Note that it's not enough to replace "delete member" by "mark member deleted in <version>". You also have to keep full history for "is it optional". And for types, because those can evolve compatibly, too, e.g. from struct to flat union, or from string to alternate of string and something else. What is the proper version-spanning interface in all the possible cases?
Quoted 3 lines
Apps that wish to have version compat, would of course need to write their code to be aware of which fields they need to seend for which QEMU version.
At which point we're reinventing libvirt.
Quoted 24 lines
Quoted 24 lines
Quoted 23 lines
* Status
There are a few rough edges to work on but this is usable. The major thing I forgot to add is handling Error from Commands. It'll be the first thing I'll work on next week.
If you want to start using this Today you can fetch it in at
https://gitlab.com/victortoso/qapi-go/
There are quite a few tests that I took from the examples in the qapi schema. Coverage using go's cover tool is giving
28.6% of statementsI've uploaded the a static generated godoc output of the above Go module here:
https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/
* License
While the generator (
golang.pyin this series) is GPL v2, the
Quoted 7 lines
Quoted 2 lines
I'd make it v2+, just to express my displeasure with the decision to make the initial QAPI generator v2 only for no good reason at all.
Our policy is that all new code should be v2+ anyway, unless it was clearly derived from some pre-existing v2-only code. Upto Victor to say whether the
golang.pyis considered clean, or was copy+paste in any parts from existin v2-only code
Makes sense.
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.
Meh. Can't be helped, I guess.
This does make me wonder though whether the license of the QAPI input files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is the Go code to be considered a derived work of the QAPI JSON files. I'm not finding a clearly articulated POV on this question so far.
Oww. You're right.
The safe and easy answer is "same license as the generator code". Anything else is either not safe or not easy, I'm afraid.
[*] Because everyhing in QEMU must be called either "device" or "driver". It's the law!
Quoted 36 lines · Markus Armbruster
Quoted 25 lines · Daniel P. Berrangé
Quoted 24 lines · Markus Armbruster
We need to look at "following the QEMU releases" a bit more closely.
Merging your patches gives us the capability to generate a Go interface to HEAD's version of QMP.
The obvious way for an out-of-tree Go program to use this generated Go interface is to build with a specific version of it. It can then talk QMP to any compatible QEMU version.
Compatibility with older QEMUs is not assured: stuff added since is present on the Go QMP client end, but not on the QEMU QMP server end.
Compatibility with newer QEMUs is subject to our deprecation policy:
In general features are intended to be supported indefinitely once introduced into QEMU. In the event that a feature needs to be removed, it will be listed in this section. The feature will remain functional for the release in which it was deprecated and one further release. After these two releases, the feature is liable to be removed.
So, if you stay away from deprecated stuff, you're good for two more releases at least.
Quoted 8 lines
Quoted 1 line
Does this work for the projects you have in mind?
It might work for some projects, but in the general case I find it pretty unappealing as a restriction. Mixing and matching new QEMU with old libvirt, or vica-verca has been an incredibly common thing todo when both developing and perhaps more importantly debugging problems. For example I have one libvirt build and I use it against any QEMU from Fedora / any RHEL-8.x update, which spans a great many QEMU releases.
Quoted 2 lines
I'd like to propose that for compatibility with a wide range of QEMU versions, you use or reinvent libvirt.
Implicit in that statement though is that libvirt will not be able to make use of the QAPI code generator as proposed though. If we are designing something to make our application consumer's lives easier, but we exclude such a major application, is our solution actually a good one.
Quoted 10 lines
Quoted 8 lines
For a minimum viable use case, this doesn't feel all that difficult, as conceptually instead of deleting the field from QAPI, we just need to annotate it to say when it was deleted from the QEMU side. The QAPI generator for internal QEMU usage, can omit any fields annotated as deleted in QAPI schema. The QAPI generator for external app usage, can (optionally) be told to include deleted fields ranging back to a given version number. So apps can chooses what degree of compat they wish to retain.
Consider this evolution of command
block_resize
To help us understand, I'll illustrate some possible interfaces in both Go and Python, since that covers dynamic and static languages
Quoted 1 line
* Initially, it has a mandatory argument @device[*].
Python definition:
def block_resize(device, size)
Caller:
block_resize('dev0', 1*GiB)
Golang definition
type BlockResizeCommand struct {
Device string
Size int
}Caller
cmd := &BlockResizeCommand{
Device: "dev0",
Size: 1 * GiB,
}Quoted 4 lines
* An alternative way to specify the command's object emerges: new argument @node-name. Both old @device and new @node-name become optional, and exactly one of them must be specified. This is commit
3b1dbd11a6"qmp: Allowblock_resizeto manipulate bs graph nodes."
Python definition. Tricky, as non-optional params must be before optional params, but size is naturally the last arg. One option is to pointlessly mark 'size' as optional
def block_resize(device=None, node_name=None, size=None)
Caller
block_resize(device="dev0", size=1*GiB)
block_resize(node_name="devnode0", size=1*GiB)In golang definition
type BlockResizeArguments struct {
Device string
NodeName string
Size int
}Caller choice of
cmd := &BlockResizeCommand{
Device: "dev0",
Size: 1 * GiB,
}
cmd := &BlockResizeCommand{
NodeName: "devnode0",
Size: 1 * GiB,
}Neither case can easily prevent passing Device and NodeName at same time.
Quoted 2 lines
* At some future date, the old way gets deprecated: argument @device acquires feature @deprecated.
Ok, no change needed to the APIs in either case. Possibly have code emit a warning if a deprecated field is set.
Quoted 2 lines
* Still later, the old way gets removed: @device is deleted, and @node-name becomes mandatory.
Again no change needed to APIs, but QEMU will throw back an error if the wrong one is used.
Quoted 5 lines
What is the proper version-spanning interface?
I figure it's both arguments optional, must specify the right one for the version of QEMU actually in use. This spans versions, but it fails to abstract from them.
Yep, I think that's inevitable in this scenario. THe plus side is that apps that want to span versions can do so. The downside is that apps that don't want smarts to span version, may loose compile time warnings about use of the now deleted field.
I suggested the code generator have an option to say what level of compat to use for generated code, so that apps can request an API without compat, which will result in compile errors. This though assumes every consumer app is embedding their own generated copy of the code. Not neccessarily desirable.
At the C level you can play games with __deprecated__ to get compile time warnings in some cases.
#define GLIB_VERSION_MIN_REQUIRED GLIB_VERSION_2_56
causes QEMU to get compile time warnings (or errors) if it attempts to use a API feature deprecated in 2.56, even if the API exists in the header & library.
Quoted 6 lines
Note that it's not enough to replace "delete member" by "mark member deleted in <version>". You also have to keep full history for "is it optional". And for types, because those can evolve compatibly, too, e.g. from struct to flat union, or from string to alternate of string and something else. What is the proper version-spanning interface in all the possible cases?
I've not thought through all possible scenarios, but there may end up being restrictions, such that changes that were previously possible may have to be forbidden.
One example, in the past we could do deprecate a field 'foo', then delete 'foo' and then some time re-introduce 'foo' with a completely different type. That would not be possible if we wanted to maintain compat, but in this example that's probably a good thing, as it'll be super confusing to have the same field name change type like that over time. Easier to just use a different name.
So the question to me is not whether all our previous changes are still possible, but whether enough of the typwes of change are possible, such that we can cope with the ongoing maint in a reasonable way. I don't think we've explored the possibility enough to say one way or the other.
Quoted 5 lines
Quoted 3 lines
Apps that wish to have version compat, would of course need to write their code to be aware of which fields they need to seend for which QEMU version.
At which point we're reinventing libvirt.
The premise of the code generators is that there are applications that want to consume QEMU that are not libvirt. With this line of reasoning we could easily say that all such applications should just use libvirt and then we don't need to provide any of these code generators. The fact that we're considering these code generators though, says that we're accepting there are valid use cases that don't want to use libvirt for whatever reasons. It is reasonable that some of those applications may wish to target a wide range of QEMU versions, just like libvirt does.
It is also reasonable to say that libvirt would be better off if it could auto-generate a client API for QEMU too, instead of writing it by hand from a human reading the QAPI
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 96 lines · Daniel P. Berrangé
Quoted 10 lines · Markus Armbruster
Quoted 8 lines
For a minimum viable use case, this doesn't feel all that difficult, as conceptually instead of deleting the field from QAPI, we just need to annotate it to say when it was deleted from the QEMU side. The QAPI generator for internal QEMU usage, can omit any fields annotated as deleted in QAPI schema. The QAPI generator for external app usage, can (optionally) be told to include deleted fields ranging back to a given version number. So apps can chooses what degree of compat they wish to retain.
Consider this evolution of command
block_resizeTo help us understand, I'll illustrate some possible interfaces in both Go and Python, since that covers dynamic and static languages
Quoted 1 line
* Initially, it has a mandatory argument @device[*].
Python definition:
def
block_resize(device, size)Caller:
block_resize('dev0', 1*GiB)Golang definition
type BlockResizeCommand struct { Device string Size int }Caller
cmd := &BlockResizeCommand{ Device: "dev0", Size: 1 * GiB, }Quoted 4 lines
* An alternative way to specify the command's object emerges: new argument @node-name. Both old @device and new @node-name become optional, and exactly one of them must be specified. This is commit
3b1dbd11a6"qmp: Allowblock_resizeto manipulate bs graph nodes."Python definition. Tricky, as non-optional params must be before optional params, but size is naturally the last arg. One option is to pointlessly mark 'size' as optional
def
block_resize(device=None, node_name=None, size=None)Caller
block_resize(device="dev0", size=1*GiB) block_resize(node_name="devnode0", size=1*GiB)In golang definition
type BlockResizeArguments struct { Device string NodeName string Size int }Caller choice of
cmd := &BlockResizeCommand{ Device: "dev0", Size: 1 * GiB, } cmd := &BlockResizeCommand{ NodeName: "devnode0", Size: 1 * GiB, }Neither case can easily prevent passing Device and NodeName at same time.
Quoted 2 lines
* At some future date, the old way gets deprecated: argument @device acquires feature @deprecated.
Ok, no change needed to the APIs in either case. Possibly have code emit a warning if a deprecated field is set.
Quoted 2 lines
* Still later, the old way gets removed: @device is deleted, and @node-name becomes mandatory.
Again no change needed to APIs, but QEMU will throw back an error if the wrong one is used.
Quoted 2 lines
What is the proper version-spanning interface?
Quoted 8 lines
Quoted 3 lines
I figure it's both arguments optional, must specify the right one for the version of QEMU actually in use. This spans versions, but it fails to abstract from them.
Yep, I think that's inevitable in this scenario. THe plus side is that apps that want to span versions can do so. The downside is that apps that don't want smarts to span version, may loose compile time warnings about use of the now deleted field.
Having said that, a different way to approach the problem is to expose the versioning directly in the generated code.
Consider a QAPI with versioning info about the fields
{ 'command': 'block_resize',
'since': '5.0.0',
'data': { 'device': ['type': 'str', 'until': '5.2.0' ],
'*device': ['type': 'str', 'since': '5.2.0', 'until': '7.0.0' ],
'*node-name': ['type': 'str', 'since': '5.2.0', 'until: '7.0.0' ],
'node-name': ['type': 'str', 'since': '7.0.0' ],
'size': 'int' } }Meaning
* Introduced in 5.0.0, with 'device' mandatory
* In 5.2.0, 'device' becomes optional, with optional 'node-name' as alternative
* In 7.0.0, 'device' is deleted, and 'node-name' becomes mandatory
Now consider the Go structs
In 5.0.0 we can generate:
type BlockResizeArguments struct {
V500 *BlockResizeArguments500
}
type BlockResizeArgumentsV1 struct {
Device string
Size int
}app can use
dev := "dev0"
cmd := BlockResizeArguments{
V500: &BlockResizeArguments500{
Device: dev,
Size: 1 * GiB
}
}In 5.2.0 we can now generate
type BlockResizeArguments struct {
V500 *BlockResizeArgumentsV500
V520 *BlockResizeArgumentsV520
}
type BlockResizeArgumentsV500 struct {
Device string
Size int
} type BlockResizeArgumentsV520 struct {
Device *string
NodeName *string
Size int
}App can use the same as before, or switch to one of
dev := "dev0"
cmd := BlockResizeArguments{
V520: &BlockResizeArguments520{
Device: &dev,
Size: 1 * GiB
}
}or
node := "nodedev0"
cmd := BlockResizeArguments{
V520: &BlockResizeArguments520{
NodeName: &node,
Size: 1 * GiB
}
}In 7.0.0 we can now generate
type BlockResizeArguments struct {
V500 *BlockResizeArgumentsV500
V520 *BlockResizeArgumentsV520
V700 *BlockResizeArgumentsV700
}
type BlockResizeArgumentsV500 struct {
Device string
Size int
} type BlockResizeArgumentsV520 struct {
Device *string
NodeName *string
Size int
}
type BlockResizeArgumentsV700 struct {
NodeName string
Size int
}
App can use the same as before, or switch to
node := "nodedev0"
cmd := BlockResizeArguments{
V700: &BlockResizeArguments700{
NodeName: node,
Size: 1 * GiB
}
}This kind of per-command/type versioning is not uncommon when defining API protocols/interfaces.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 228 lines · Daniel P. Berrangé
Quoted 96 lines · Daniel P. Berrangé
Quoted 10 lines · Markus Armbruster
Quoted 8 lines
For a minimum viable use case, this doesn't feel all that difficult, as conceptually instead of deleting the field from QAPI, we just need to annotate it to say when it was deleted from the QEMU side. The QAPI generator for internal QEMU usage, can omit any fields annotated as deleted in QAPI schema. The QAPI generator for external app usage, can (optionally) be told to include deleted fields ranging back to a given version number. So apps can chooses what degree of compat they wish to retain.
Consider this evolution of command
block_resizeTo help us understand, I'll illustrate some possible interfaces in both Go and Python, since that covers dynamic and static languages
Quoted 1 line
* Initially, it has a mandatory argument @device[*].
Python definition:
def
block_resize(device, size)Caller:
block_resize('dev0', 1*GiB)Golang definition
type BlockResizeCommand struct { Device string Size int }Caller
cmd := &BlockResizeCommand{ Device: "dev0", Size: 1 * GiB, }Quoted 4 lines
* An alternative way to specify the command's object emerges: new argument @node-name. Both old @device and new @node-name become optional, and exactly one of them must be specified. This is commit
3b1dbd11a6"qmp: Allowblock_resizeto manipulate bs graph nodes."Python definition. Tricky, as non-optional params must be before optional params, but size is naturally the last arg. One option is to pointlessly mark 'size' as optional
def
block_resize(device=None, node_name=None, size=None)Caller
block_resize(device="dev0", size=1*GiB) block_resize(node_name="devnode0", size=1*GiB)In golang definition
type BlockResizeArguments struct { Device string NodeName string Size int }Caller choice of
cmd := &BlockResizeCommand{ Device: "dev0", Size: 1 * GiB, } cmd := &BlockResizeCommand{ NodeName: "devnode0", Size: 1 * GiB, }Neither case can easily prevent passing Device and NodeName at same time.
Quoted 2 lines
* At some future date, the old way gets deprecated: argument @device acquires feature @deprecated.
Ok, no change needed to the APIs in either case. Possibly have code emit a warning if a deprecated field is set.
Quoted 2 lines
* Still later, the old way gets removed: @device is deleted, and @node-name becomes mandatory.
Again no change needed to APIs, but QEMU will throw back an error if the wrong one is used.
Quoted 2 lines
What is the proper version-spanning interface?
Quoted 8 lines
Quoted 3 lines
I figure it's both arguments optional, must specify the right one for the version of QEMU actually in use. This spans versions, but it fails to abstract from them.
Yep, I think that's inevitable in this scenario. THe plus side is that apps that want to span versions can do so. The downside is that apps that don't want smarts to span version, may loose compile time warnings about use of the now deleted field.
Having said that, a different way to approach the problem is to expose the versioning directly in the generated code.
Consider a QAPI with versioning info about the fields
{ 'command': '
block_resize','since': '5.0.0', 'data': { 'device': ['type': 'str', 'until': '5.2.0' ], '*device': ['type': 'str', 'since': '5.2.0', 'until': '7.0.0' ], '*node-name': ['type': 'str', 'since': '5.2.0', 'until: '7.0.0' ], 'node-name': ['type': 'str', 'since': '7.0.0' ], 'size': 'int' } }Meaning
* Introduced in 5.0.0, with 'device' mandatory
* In 5.2.0, 'device' becomes optional, with optional 'node-name' as alternative
* In 7.0.0, 'device' is deleted, and 'node-name' becomes mandatoryNow consider the Go structs
In 5.0.0 we can generate:
type BlockResizeArguments struct { V500 *BlockResizeArguments500 } type BlockResizeArgumentsV1 struct { Device string Size int }app can use
dev := "dev0" cmd := BlockResizeArguments{ V500: &BlockResizeArguments500{ Device: dev, Size: 1 * GiB } }In 5.2.0 we can now generate
type BlockResizeArguments struct { V500 *BlockResizeArgumentsV500 V520 *BlockResizeArgumentsV520 } type BlockResizeArgumentsV500 struct { Device string Size int }type BlockResizeArgumentsV520 struct { Device *string NodeName *string Size int }App can use the same as before, or switch to one of
dev := "dev0" cmd := BlockResizeArguments{ V520: &BlockResizeArguments520{ Device: &dev, Size: 1 * GiB } }or
node := "nodedev0" cmd := BlockResizeArguments{ V520: &BlockResizeArguments520{ NodeName: &node, Size: 1 * GiB } }In 7.0.0 we can now generate
type BlockResizeArguments struct { V500 *BlockResizeArgumentsV500 V520 *BlockResizeArgumentsV520 V700 *BlockResizeArgumentsV700 } type BlockResizeArgumentsV500 struct { Device string Size int }type BlockResizeArgumentsV520 struct { Device *string NodeName *string Size int } type BlockResizeArgumentsV700 struct { NodeName string Size int }App can use the same as before, or switch to
node := "nodedev0" cmd := BlockResizeArguments{ V700: &BlockResizeArguments700{ NodeName: node, Size: 1 * GiB } }
Quoted 2 lines
This kind of per-command/type versioning is not uncommon when defining API protocols/interfaces.
1. At every release, put a copy of the QAPI schema in the freezer.
2. For every copy, generate Go types with a suitable name suffix. Collect all the name stems.
3. For each name stem, define a Go struct that contains all the suffixed Go types with that stem.
Look Ma, no cluttering the QAPI schema with a full history! Also no complicating the schema language to provide the means for that.
Quoted 130 lines · Markus Armbruster
Quoted 123 lines · Daniel P. Berrangé
On Tue, May 10, 2022 at 01:34:03PM +0100, Daniel P. Berrangé wrote:
Having said that, a different way to approach the problem is to expose the versioning directly in the generated code.Consider a QAPI with versioning info about the fields
{ 'command': '
block_resize','since': '5.0.0', 'data': { 'device': ['type': 'str', 'until': '5.2.0' ], '*device': ['type': 'str', 'since': '5.2.0', 'until': '7.0.0' ], '*node-name': ['type': 'str', 'since': '5.2.0', 'until: '7.0.0' ], 'node-name': ['type': 'str', 'since': '7.0.0' ], 'size': 'int' } }Meaning
* Introduced in 5.0.0, with 'device' mandatory
* In 5.2.0, 'device' becomes optional, with optional 'node-name' as alternative
* In 7.0.0, 'device' is deleted, and 'node-name' becomes mandatoryNow consider the Go structs
In 5.0.0 we can generate:
type BlockResizeArguments struct { V500 *BlockResizeArguments500 } type BlockResizeArgumentsV1 struct { Device string Size int }app can use
dev := "dev0" cmd := BlockResizeArguments{ V500: &BlockResizeArguments500{ Device: dev, Size: 1 * GiB } }In 5.2.0 we can now generate
type BlockResizeArguments struct { V500 *BlockResizeArgumentsV500 V520 *BlockResizeArgumentsV520 } type BlockResizeArgumentsV500 struct { Device string Size int }type BlockResizeArgumentsV520 struct { Device *string NodeName *string Size int }App can use the same as before, or switch to one of
dev := "dev0" cmd := BlockResizeArguments{ V520: &BlockResizeArguments520{ Device: &dev, Size: 1 * GiB } }or
node := "nodedev0" cmd := BlockResizeArguments{ V520: &BlockResizeArguments520{ NodeName: &node, Size: 1 * GiB } }In 7.0.0 we can now generate
type BlockResizeArguments struct { V500 *BlockResizeArgumentsV500 V520 *BlockResizeArgumentsV520 V700 *BlockResizeArgumentsV700 } type BlockResizeArgumentsV500 struct { Device string Size int }type BlockResizeArgumentsV520 struct { Device *string NodeName *string Size int } type BlockResizeArgumentsV700 struct { NodeName string Size int }App can use the same as before, or switch to
node := "nodedev0" cmd := BlockResizeArguments{ V700: &BlockResizeArguments700{ NodeName: node, Size: 1 * GiB } }Quoted 2 lines
This kind of per-command/type versioning is not uncommon when defining API protocols/interfaces.
1. At every release, put a copy of the QAPI schema in the freezer.
Quoted 8 lines
2. For every copy, generate Go types with a suitable name suffix. Collect all the name stems.
3. For each name stem, define a Go struct that contains all the suffixed Go types with that stem.
Look Ma, no cluttering the QAPI schema with a full history! Also no complicating the schema language to provide the means for that.
That could indeed be a viable approach. Puts a little more work on the code generator to match up the types, but probably isn't too hard.
Incidentally, we've intentionally not exposed type names in the QAPI introspection in QMP. With code generators, when the generated code type names driven from QAPI schema, there's likely going to be an expectation that type names in QAPI have some kind of stability rules.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 26 lines · Daniel P. Berrangé
In 7.0.0 we can now generate
type BlockResizeArguments struct { V500 *BlockResizeArgumentsV500 V520 *BlockResizeArgumentsV520 V700 *BlockResizeArgumentsV700 } type BlockResizeArgumentsV500 struct { Device string Size int } type BlockResizeArgumentsV520 struct { Device *string NodeName *string Size int } type BlockResizeArgumentsV700 struct { NodeName string Size int }App can use the same as before, or switch to
Quoted 7 lines
node := "nodedev0"
cmd := BlockResizeArguments{
V700: &BlockResizeArguments700{
NodeName: node,
Size: 1 * GiB
}
}This honestly looks pretty unwieldy.
If the application already knows it's targeting a specific version of the QEMU API, which for the above code to make any sense it will have to, couldn't it do something like
import qemu .../qemu/v700
at the beginning of the file and then use regular old
cmd := qemu.BlockResizeArguments{
NodeName: nodeName,
Size: size,
}instead?
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Quoted 35 lines · Andrea Bolognani
Quoted 26 lines · Daniel P. Berrangé
In 7.0.0 we can now generate
type BlockResizeArguments struct { V500 *BlockResizeArgumentsV500 V520 *BlockResizeArgumentsV520 V700 *BlockResizeArgumentsV700 } type BlockResizeArgumentsV500 struct { Device string Size int } type BlockResizeArgumentsV520 struct { Device *string NodeName *string Size int } type BlockResizeArgumentsV700 struct { NodeName string Size int }App can use the same as before, or switch to
Quoted 7 lines
node := "nodedev0" cmd := BlockResizeArguments{ V700: &BlockResizeArguments700{ NodeName: node, Size: 1 * GiB } }
Quoted 1 line
This honestly looks pretty unwieldy.
It isn't all that more verbose than without the versions - just a single struct wrapper.
Quoted 7 lines
If the application already knows it's targeting a specific version of the QEMU API, which for the above code to make any sense it will have to, couldn't it do something like
import qemu .../qemu/v700
Quoted 8 lines
at the beginning of the file and then use regular old
cmd := qemu.BlockResizeArguments{ NodeName: nodeName, Size: size, }instead?
This would lead to a situation where every struct is duplicated for every version, even though 90% of the time they'll be identical across multiple versions. This is not very ammenable to the desire to be able to dynamically choose per-command which version you want based on which version of QEMU you're connected to.
ie
var cmd Command
if qmp.HasVersion(qemu.Version(7, 0, 0)) {
cmd = BlockResizeArguments{
V700: &BlockResizeArguments700{
NodeName: node,
Size: 1 * GiB
}
}
} else {
cmd = BlockResizeArguments{
V520: &BlockResizeArguments520{
Device: dev,
Size: 1 * GiB
}
}
}And of course the HasVersion check is going to be different for each command that matters.
Having said that, this perhaps shows the nested structs are overkill. We could have
var cmd Command
if qmp.HasVersion(qemu.Version(7, 0, 0)) {
cmd = &BlockResizeArguments700{
NodeName: node,
Size: 1 * GiB
}
} else {
cmd = &BlockResizeArguments520{
Device: dev,
Size: 1 * GiB
}
}If there was some need for common handling of the different versioned variants, we could still have a 'BlockResizeArguments' that has a field per version, as an optional thing. Or have a BlockResizeArguments interface, implemented by each version </hand-wavey>
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Quoted 31 lines · Daniel P. Berrangé
This would lead to a situation where every struct is duplicated for every version, even though 90% of the time they'll be identical across multiple versions. This is not very ammenable to the desire to be able to dynamically choose per-command which version you want based on which version of QEMU you're connected to.
ie
var cmd Command if qmp.HasVersion(qemu.Version(7, 0, 0)) { cmd = BlockResizeArguments{ V700: &BlockResizeArguments700{ NodeName: node, Size: 1 * GiB } } } else { cmd = BlockResizeArguments{ V520: &BlockResizeArguments520{ Device: dev, Size: 1 * GiB } } }And of course the HasVersion check is going to be different for each command that matters.
Having said that, this perhaps shows the nested structs are overkill. We could have
Quoted 12 lines
var cmd Command
if qmp.HasVersion(qemu.Version(7, 0, 0)) {
cmd = &BlockResizeArguments700{
NodeName: node,
Size: 1 * GiB
}
} else {
cmd = &BlockResizeArguments520{
Device: dev,
Size: 1 * GiB
}
}Right, making the decision at the import level would require adding a level of indirection and make this kind of dynamic logic awkward.
We shouldn't force users to sprinkle version numbers everywhere though, especially since different commands will change at different points in time. It should be possible to do something like
if !qmp.HasAPI(600) {
panic("update QEMU")
}
cmd := &BlockResizeArguments600{ // really BlockResizeArguments520
Device: device,
Size: size,
}or even
if !qmp.HasAPI(qmp.API.Latest) {
panic("update QEMU")
}
cmd := &BlockResizeArguments{
NodeName: nodeName,
Size: size,
}Should be easy enough to achieve with type aliases.
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Quoted 72 lines · Andrea Bolognani
Quoted 31 lines · Daniel P. Berrangé
This would lead to a situation where every struct is duplicated for every version, even though 90% of the time they'll be identical across multiple versions. This is not very ammenable to the desire to be able to dynamically choose per-command which version you want based on which version of QEMU you're connected to.
ie
var cmd Command if qmp.HasVersion(qemu.Version(7, 0, 0)) { cmd = BlockResizeArguments{ V700: &BlockResizeArguments700{ NodeName: node, Size: 1 * GiB } } } else { cmd = BlockResizeArguments{ V520: &BlockResizeArguments520{ Device: dev, Size: 1 * GiB } } }And of course the HasVersion check is going to be different for each command that matters.
Having said that, this perhaps shows the nested structs are overkill. We could have
Quoted 12 lines
var cmd Command if qmp.HasVersion(qemu.Version(7, 0, 0)) { cmd = &BlockResizeArguments700{ NodeName: node, Size: 1 * GiB } } else { cmd = &BlockResizeArguments520{ Device: dev, Size: 1 * GiB } }Right, making the decision at the import level would require adding a level of indirection and make this kind of dynamic logic awkward.
We shouldn't force users to sprinkle version numbers everywhere though, especially since different commands will change at different points in time. It should be possible to do something like
if !qmp.HasAPI(600) { panic("update QEMU") } cmd := &BlockResizeArguments600{ // really BlockResizeArguments520 Device: device, Size: size, }or even
if !qmp.HasAPI(qmp.API.Latest) { panic("update QEMU") } cmd := &BlockResizeArguments{ NodeName: nodeName, Size: size, }
Quoted 1 line
Should be easy enough to achieve with type aliases.
I guess we would have a single package which does
typedef BlockResizeArguments520 BlockResizeArguments; ...for each type...
The interaction with API versioning will be tedious though. For the versioned structs we'll be forever back compatible, due to the append-only nature of the versioned struct approach. For the type aliases, we'll be forever breaking compat as at least 1 struct alias is likely to need changing every QEMU release.
Might suggest having 2 distinct go modules. A base module which is versioned as a stable API with semver (forever v1.xxx), and an add-on module that depends on base module, which is versioned as an unstable API with semver (forever v0.xxx)
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Hi,
Quoted 90 lines · Daniel P. Berrangé
Quoted 35 lines · Andrea Bolognani
Quoted 26 lines · Daniel P. Berrangé
In 7.0.0 we can now generate
type BlockResizeArguments struct { V500 *BlockResizeArgumentsV500 V520 *BlockResizeArgumentsV520 V700 *BlockResizeArgumentsV700 } type BlockResizeArgumentsV500 struct { Device string Size int } type BlockResizeArgumentsV520 struct { Device *string NodeName *string Size int } type BlockResizeArgumentsV700 struct { NodeName string Size int }App can use the same as before, or switch to
Quoted 7 lines
node := "nodedev0" cmd := BlockResizeArguments{ V700: &BlockResizeArguments700{ NodeName: node, Size: 1 * GiB } }Quoted 1 line
This honestly looks pretty unwieldy.
It isn't all that more verbose than without the versions - just a single struct wrapper.
Quoted 7 lines
If the application already knows it's targeting a specific version of the QEMU API, which for the above code to make any sense it will have to, couldn't it do something like
import qemu .../qemu/v700
Quoted 8 lines
at the beginning of the file and then use regular old
cmd := qemu.BlockResizeArguments{ NodeName: nodeName, Size: size, }instead?
This would lead to a situation where every struct is duplicated for every version, even though 90% of the time they'll be identical across multiple versions. This is not very ammenable to the desire to be able to dynamically choose per-command which version you want based on which version of QEMU you're connected to.
ie
var cmd Command if qmp.HasVersion(qemu.Version(7, 0, 0)) { cmd = BlockResizeArguments{ V700: &BlockResizeArguments700{ NodeName: node, Size: 1 * GiB } } } else { cmd = BlockResizeArguments{ V520: &BlockResizeArguments520{ Device: dev, Size: 1 * GiB } } }And of course the HasVersion check is going to be different for each command that matters.
Having said that, this perhaps shows the nested structs are overkill. We could have
Quoted 12 lines
var cmd Command
if qmp.HasVersion(qemu.Version(7, 0, 0)) {
cmd = &BlockResizeArguments700{
NodeName: node,
Size: 1 * GiB
}
} else {
cmd = &BlockResizeArguments520{
Device: dev,
Size: 1 * GiB
}
}The else block would be wrong in versions above 7.0.0 where block_resize changed. There will be a need to know for a specific Type if we are covered with latest qemu/qapi-go or not. Not yet sure how to address that, likely we will need to keep the information that something has been added/changed/removed per version per Type in qapi-go...
Still, I think the above proposal is a good compromise to make..
Quoted 4 lines
If there was some need for common handling of the different versioned variants, we could still have a 'BlockResizeArguments' that has a field per version, as an optional thing. Or have a BlockResizeArguments interface, implemented by each version </hand-wavey>
Cheers,
Victor
Quoted 106 lines · Victor Toso
Hi,
Quoted 90 lines · Daniel P. Berrangé
Quoted 35 lines · Andrea Bolognani
Quoted 26 lines · Daniel P. Berrangé
In 7.0.0 we can now generate
type BlockResizeArguments struct { V500 *BlockResizeArgumentsV500 V520 *BlockResizeArgumentsV520 V700 *BlockResizeArgumentsV700 } type BlockResizeArgumentsV500 struct { Device string Size int } type BlockResizeArgumentsV520 struct { Device *string NodeName *string Size int } type BlockResizeArgumentsV700 struct { NodeName string Size int }App can use the same as before, or switch to
Quoted 7 lines
node := "nodedev0" cmd := BlockResizeArguments{ V700: &BlockResizeArguments700{ NodeName: node, Size: 1 * GiB } }Quoted 1 line
This honestly looks pretty unwieldy.
It isn't all that more verbose than without the versions - just a single struct wrapper.
Quoted 7 lines
If the application already knows it's targeting a specific version of the QEMU API, which for the above code to make any sense it will have to, couldn't it do something like
import qemu .../qemu/v700
Quoted 8 lines
at the beginning of the file and then use regular old
cmd := qemu.BlockResizeArguments{ NodeName: nodeName, Size: size, }instead?
This would lead to a situation where every struct is duplicated for every version, even though 90% of the time they'll be identical across multiple versions. This is not very ammenable to the desire to be able to dynamically choose per-command which version you want based on which version of QEMU you're connected to.
ie
var cmd Command if qmp.HasVersion(qemu.Version(7, 0, 0)) { cmd = BlockResizeArguments{ V700: &BlockResizeArguments700{ NodeName: node, Size: 1 * GiB } } } else { cmd = BlockResizeArguments{ V520: &BlockResizeArguments520{ Device: dev, Size: 1 * GiB } } }And of course the HasVersion check is going to be different for each command that matters.
Having said that, this perhaps shows the nested structs are overkill. We could have
Quoted 12 lines
var cmd Command if qmp.HasVersion(qemu.Version(7, 0, 0)) { cmd = &BlockResizeArguments700{ NodeName: node, Size: 1 * GiB } } else { cmd = &BlockResizeArguments520{ Device: dev, Size: 1 * GiB } }
Quoted 6 lines
The else block would be wrong in versions above 7.0.0 where
block_resizechanged. There will be a need to know for a specific Type if we are covered with latest qemu/qapi-go or not. Not yet sure how to address that, likely we will need to keep the information that something has been added/changed/removed per version per Type in qapi-go...
I put that in the "nice to have" category. No application can predict the future, and nor do they really need to try in general.
If the application code was written when the newest QEMU was 7.1.0, and the above code is correct for QEMU <= 7.1.0, then that's good enough. If the BlockResizeArguments struct changed in a later QEMU version 8.0.0, that doesn't matter at the point the app code was written.
Much of the time the changes are back compatible, ie just adding a new field, and so everything will still work fine if the app carries on using BlockResizeArguments700, despite a new BlockResizeArguments800 arriving with a new field.
Only in the cases where a field was removed or changed in a non-compatible manner would an app have problems, and QEMU will happily report an error at runtime if the app sends something incompatible.
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
HI,
Quoted 118 lines · Daniel P. Berrangé
Quoted 106 lines · Victor Toso
Hi,
Quoted 90 lines · Daniel P. Berrangé
Quoted 35 lines · Andrea Bolognani
Quoted 26 lines · Daniel P. Berrangé
In 7.0.0 we can now generate
type BlockResizeArguments struct { V500 *BlockResizeArgumentsV500 V520 *BlockResizeArgumentsV520 V700 *BlockResizeArgumentsV700 } type BlockResizeArgumentsV500 struct { Device string Size int } type BlockResizeArgumentsV520 struct { Device *string NodeName *string Size int } type BlockResizeArgumentsV700 struct { NodeName string Size int }App can use the same as before, or switch to
Quoted 7 lines
node := "nodedev0" cmd := BlockResizeArguments{ V700: &BlockResizeArguments700{ NodeName: node, Size: 1 * GiB } }Quoted 1 line
This honestly looks pretty unwieldy.
It isn't all that more verbose than without the versions - just a single struct wrapper.
Quoted 7 lines
If the application already knows it's targeting a specific version of the QEMU API, which for the above code to make any sense it will have to, couldn't it do something like
import qemu .../qemu/v700
Quoted 8 lines
at the beginning of the file and then use regular old
cmd := qemu.BlockResizeArguments{ NodeName: nodeName, Size: size, }instead?
This would lead to a situation where every struct is duplicated for every version, even though 90% of the time they'll be identical across multiple versions. This is not very ammenable to the desire to be able to dynamically choose per-command which version you want based on which version of QEMU you're connected to.
ie
var cmd Command if qmp.HasVersion(qemu.Version(7, 0, 0)) { cmd = BlockResizeArguments{ V700: &BlockResizeArguments700{ NodeName: node, Size: 1 * GiB } } } else { cmd = BlockResizeArguments{ V520: &BlockResizeArguments520{ Device: dev, Size: 1 * GiB } } }And of course the HasVersion check is going to be different for each command that matters.
Having said that, this perhaps shows the nested structs are overkill. We could have
Quoted 12 lines
var cmd Command if qmp.HasVersion(qemu.Version(7, 0, 0)) { cmd = &BlockResizeArguments700{ NodeName: node, Size: 1 * GiB } } else { cmd = &BlockResizeArguments520{ Device: dev, Size: 1 * GiB } }Quoted 6 lines
The else block would be wrong in versions above 7.0.0 where
block_resizechanged. There will be a need to know for a specific Type if we are covered with latest qemu/qapi-go or not. Not yet sure how to address that, likely we will need to keep the information that something has been added/changed/removed per version per Type in qapi-go...I put that in the "nice to have" category. No application can predict the future, and nor do they really need to try in general.
Quoted 5 lines
If the application code was written when the newest QEMU was 7.1.0, and the above code is correct for QEMU <= 7.1.0, then that's good enough. If the BlockResizeArguments struct changed in a later QEMU version 8.0.0, that doesn't matter at the point the app code was written.
I'm not thinking at runtime, I'm thinking at compile time.
I update $myproject's qpai-go to 8.0.0 and get a warnings that some types would not work with 8.0.0 (e.g: because there is a new BlockResizeArguments800)
Quoted 9 lines
Much of the time the changes are back compatible, ie just adding a new field, and so everything will still work fine if the app carries on using BlockResizeArguments700, despite a new BlockResizeArguments800 arriving with a new field.
Only in the cases where a field was removed or changed in a non-compatible manner would an app have problems, and QEMU will happily report an error at runtime if the app sends something incompatible.
Yeah, runtime error is fine but if we can provide some extra checks at the time someone wants to move qapi-go from 7.2.0 to 8.0.0, that would be great.
Cheers,
Victor
Quoted 37 lines · Daniel P. Berrangé
Quoted 36 lines · Markus Armbruster
Quoted 33 lines · Daniel P. Berrangé
Quoted 21 lines · Markus Armbruster
We need to look at "following the QEMU releases" a bit more closely.
Merging your patches gives us the capability to generate a Go interface to HEAD's version of QMP.
The obvious way for an out-of-tree Go program to use this generated Go interface is to build with a specific version of it. It can then talk QMP to any compatible QEMU version.
Compatibility with older QEMUs is not assured: stuff added since is present on the Go QMP client end, but not on the QEMU QMP server end.
Compatibility with newer QEMUs is subject to our deprecation policy:
In general features are intended to be supported indefinitely once introduced into QEMU. In the event that a feature needs to be removed, it will be listed in this section. The feature will remain functional for the release in which it was deprecated and one further release. After these two releases, the feature is liable to be removed.
Quoted 4 lines
So, if you stay away from deprecated stuff, you're good for two more releases at least.
Does this work for the projects you have in mind?
It might work for some projects, but in the general case I find it pretty unappealing as a restriction. Mixing and matching new QEMU with old libvirt, or vica-verca has been an incredibly common thing todo when both developing and perhaps more importantly debugging problems. For example I have one libvirt build and I use it against any QEMU from Fedora / any RHEL-8.x update, which spans a great many QEMU releases.
Quoted 8 lines
Quoted 2 lines
I'd like to propose that for compatibility with a wide range of QEMU versions, you use or reinvent libvirt.
Implicit in that statement though is that libvirt will not be able to make use of the QAPI code generator as proposed though. If we are designing something to make our application consumer's lives easier, but we exclude such a major application, is our solution actually a good one.
A solution for a narrow problem we can actually deliver and maintain is better than a solution for a wider problem we can't.
Quoted 45 lines
Quoted 10 lines
Quoted 8 lines
For a minimum viable use case, this doesn't feel all that difficult, as conceptually instead of deleting the field from QAPI, we just need to annotate it to say when it was deleted from the QEMU side. The QAPI generator for internal QEMU usage, can omit any fields annotated as deleted in QAPI schema. The QAPI generator for external app usage, can (optionally) be told to include deleted fields ranging back to a given version number. So apps can chooses what degree of compat they wish to retain.
Consider this evolution of command
block_resizeTo help us understand, I'll illustrate some possible interfaces in both Go and Python, since that covers dynamic and static languages
Quoted 1 line
* Initially, it has a mandatory argument @device[*].
Python definition:
def
block_resize(device, size)Caller:
block_resize('dev0', 1*GiB)Golang definition
type BlockResizeCommand struct { Device string Size int }Caller
cmd := &BlockResizeCommand{ Device: "dev0", Size: 1 * GiB, }Quoted 4 lines
* An alternative way to specify the command's object emerges: new argument @node-name. Both old @device and new @node-name become optional, and exactly one of them must be specified. This is commit
3b1dbd11a6"qmp: Allowblock_resizeto manipulate bs graph nodes."
Quoted 5 lines
Python definition. Tricky, as non-optional params must be before optional params, but size is naturally the last arg. One option is to pointlessly mark 'size' as optional
def
block_resize(device=None, node_name=None, size=None)
Who needs compile-time checking anyway.
Back to serious. Keyword arguments might be a better match for Python bindings.
Quoted 16 lines
Caller
block_resize(device="dev0", size=1*GiB) block_resize(node_name="devnode0", size=1*GiB)In golang definition
type BlockResizeArguments struct { Device string NodeName string Size int }Caller choice of
Quoted 9 lines
cmd := &BlockResizeCommand{
Device: "dev0",
Size: 1 * GiB,
}
cmd := &BlockResizeCommand{
NodeName: "devnode0",
Size: 1 * GiB,
}Note that the Go bindings you sketched effectively use (poor man's) keyword arguments.
Quoted 2 lines
Neither case can easily prevent passing Device and NodeName at same time.
That defect lies at the schema's feet.
Quoted 14 lines
Quoted 2 lines
* At some future date, the old way gets deprecated: argument @device acquires feature @deprecated.
Ok, no change needed to the APIs in either case. Possibly have code emit a warning if a deprecated field is set.
Quoted 2 lines
* Still later, the old way gets removed: @device is deleted, and @node-name becomes mandatory.
Again no change needed to APIs, but QEMU will throw back an error if the wrong one is used.
Quoted 2 lines
What is the proper version-spanning interface?
Quoted 8 lines
Quoted 3 lines
I figure it's both arguments optional, must specify the right one for the version of QEMU actually in use. This spans versions, but it fails to abstract from them.
Yep, I think that's inevitable in this scenario. THe plus side is that apps that want to span versions can do so. The downside is that apps that don't want smarts to span version, may loose compile time warnings about use of the now deleted field.
The version-spanning interface will arguably be a bad interface for any version.
Quoted 23 lines
I suggested the code generator have an option to say what level of compat to use for generated code, so that apps can request an API without compat, which will result in compile errors. This though assumes every consumer app is embedding their own generated copy of the code. Not neccessarily desirable.
At the C level you can play games with
__deprecated__to get compile time warnings in some cases.#define
GLIB_VERSION_MIN_REQUIREDGLIB_VERSION_2_56causes QEMU to get compile time warnings (or errors) if it attempts to use a API feature deprecated in 2.56, even if the API exists in the header & library.
Quoted 6 lines
Note that it's not enough to replace "delete member" by "mark member deleted in <version>". You also have to keep full history for "is it optional". And for types, because those can evolve compatibly, too, e.g. from struct to flat union, or from string to alternate of string and something else. What is the proper version-spanning interface in all the possible cases?
Quoted 3 lines
I've not thought through all possible scenarios, but there may end up being restrictions, such that changes that were previously possible may have to be forbidden.
"There may be restrictions" is not exactly a confidence-inspring design assumption. We need a reasonably dependable idea on what exactly we're intending to sacrifice.
Quoted 19 lines
One example, in the past we could do deprecate a field 'foo', then delete 'foo' and then some time re-introduce 'foo' with a completely different type. That would not be possible if we wanted to maintain compat, but in this example that's probably a good thing, as it'll be super confusing to have the same field name change type like that over time. Easier to just use a different name.
So the question to me is not whether all our previous changes are still possible, but whether enough of the typwes of change are possible, such that we can cope with the ongoing maint in a reasonable way. I don't think we've explored the possibility enough to say one way or the other.
Quoted 5 lines
Quoted 3 lines
Apps that wish to have version compat, would of course need to write their code to be aware of which fields they need to seend for which QEMU version.
At which point we're reinventing libvirt.
Quoted 7 lines
The premise of the code generators is that there are applications that want to consume QEMU that are not libvirt. With this line of reasoning we could easily say that all such applications should just use libvirt and then we don't need to provide any of these code generators. The fact that we're considering these code generators though, says that we're accepting there are valid use cases that don't want to use libvirt for whatever reasons.
Can't resist any longer: "What has libvirt ever done for us?"
https://www.youtube.com/watch?v=Qc7HmhrgTuQ
Quoted 3 lines
It is
reasonable that some of those applications may wish to target a wide range of QEMU versions, just like libvirt does.
At which point they're comitting to reinventing the relevant parts of libvirt.
I'd expect marshalling and umarshalling QMP to be among the smaller sub-problems then. It may look relatively large at first, because it's among the first ones to be solved. More so when you hand-wave away the more interesting ones of *abstraction* until they bite you in the posterior.
Quoted 6 lines
It is also reasonable to say that libvirt would be better off if it could auto-generate a client API for QEMU too, instead of writing it by hand from a human reading the QAPI
With regards,
Daniel
Hi,
Quoted 27 lines · Markus Armbruster
Quoted 16 lines · Daniel P. Berrangé
Caller
block_resize(device="dev0", size=1*GiB) block_resize(node_name="devnode0", size=1*GiB)In golang definition
type BlockResizeArguments struct { Device string NodeName string Size int }Caller choice of
Quoted 9 lines
cmd := &BlockResizeCommand{ Device: "dev0", Size: 1 * GiB, } cmd := &BlockResizeCommand{ NodeName: "devnode0", Size: 1 * GiB, }
Quoted 7 lines
Note that the Go bindings you sketched effectively use (poor man's) keyword arguments.
Quoted 2 lines
Neither case can easily prevent passing Device and NodeName at same time.
That defect lies at the schema's feet.
Right. The schema does not provide any metadata to explicit say that only @device or @node-name should be used, correct?
This would be important to differentiate of a simple 'adding a new optional argument' plus 'making this other argument optional'.
Quoted 49 lines
Quoted 14 lines
Quoted 2 lines
* At some future date, the old way gets deprecated: argument @device acquires feature @deprecated.
Ok, no change needed to the APIs in either case. Possibly have code emit a warning if a deprecated field is set.
Quoted 2 lines
* Still later, the old way gets removed: @device is deleted, and @node-name becomes mandatory.
Again no change needed to APIs, but QEMU will throw back an error if the wrong one is used.
Quoted 2 lines
What is the proper version-spanning interface?
Quoted 8 lines
Quoted 3 lines
I figure it's both arguments optional, must specify the right one for the version of QEMU actually in use. This spans versions, but it fails to abstract from them.
Yep, I think that's inevitable in this scenario. THe plus side is that apps that want to span versions can do so. The downside is that apps that don't want smarts to span version, may loose compile time warnings about use of the now deleted field.
The version-spanning interface will arguably be a bad interface for any version.
Quoted 23 lines
I suggested the code generator have an option to say what level of compat to use for generated code, so that apps can request an API without compat, which will result in compile errors. This though assumes every consumer app is embedding their own generated copy of the code. Not neccessarily desirable.
At the C level you can play games with
__deprecated__to get compile time warnings in some cases.#define
GLIB_VERSION_MIN_REQUIREDGLIB_VERSION_2_56causes QEMU to get compile time warnings (or errors) if it attempts to use a API feature deprecated in 2.56, even if the API exists in the header & library.
Quoted 6 lines
Note that it's not enough to replace "delete member" by "mark member deleted in <version>". You also have to keep full history for "is it optional". And for types, because those can evolve compatibly, too, e.g. from struct to flat union, or from string to alternate of string and something else. What is the proper version-spanning interface in all the possible cases?
Quoted 7 lines
Quoted 3 lines
I've not thought through all possible scenarios, but there may end up being restrictions, such that changes that were previously possible may have to be forbidden.
"There may be restrictions" is not exactly a confidence-inspring design assumption. We need a reasonably dependable idea on what exactly we're intending to sacrifice.
I can't help much here but I guess we can evolve QAPI schema as we move forward. Adding metadata that helps document changes to the benefit of giving code generators tools to provide a way to work with those QAPI changes seems desirable, no?
Quoted 13 lines
Quoted 13 lines
One example, in the past we could do deprecate a field 'foo', then delete 'foo' and then some time re-introduce 'foo' with a completely different type. That would not be possible if we wanted to maintain compat, but in this example that's probably a good thing, as it'll be super confusing to have the same field name change type like that over time. Easier to just use a different name.
So the question to me is not whether all our previous changes are still possible, but whether enough of the typwes of change are possible, such that we can cope with the ongoing maint in a reasonable way. I don't think we've explored the possibility enough to say one way or the other.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Apps that wish to have version compat, would of course need to write their code to be aware of which fields they need to seend for which QEMU version.
At which point we're reinventing libvirt.
IMHO, at this moment, qapi-go is targeting communicating with QEMU and handling multiple QEMU versions seems reasonable to me.
Perhaps libvirt can use qapi-go in the future or other generated interface. That would be cool.
Quoted 15 lines
Quoted 7 lines
The premise of the code generators is that there are applications that want to consume QEMU that are not libvirt. With this line of reasoning we could easily say that all such applications should just use libvirt and then we don't need to provide any of these code generators. The fact that we're considering these code generators though, says that we're accepting there are valid use cases that don't want to use libvirt for whatever reasons.
Can't resist any longer: "What has libvirt ever done for us?"
https://www.youtube.com/watch?v=Qc7HmhrgTuQQuoted 3 lines
It is
reasonable that some of those applications may wish to target a wide range of QEMU versions, just like libvirt does.
Quoted 8 lines
At which point they're comitting to reinventing the relevant parts of libvirt.
I'd expect marshalling and umarshalling QMP to be among the smaller sub-problems then. It may look relatively large at first, because it's among the first ones to be solved. More so when you hand-wave away the more interesting ones of *abstraction* until they bite you in the posterior.
I might have missed it but I don't see unsolvable problems.
1) We decide if we want a Golang interface that can communicate
with multiple QEMU versions or not;
2) We discuss how that Golang interface would look like;
3) We discuss what is needed in QEMU/QAPI to achieve (2)
4) We work on QEMU/QAPI to address (3)
5) We move on with qapi-go proposal
I see only benefits with this project, plus the fact we already have Golang projects doing their own code to communicate with QEMU and I do believe we will make their lives easier.
Quoted 6 lines
Quoted 6 lines
It is also reasonable to say that libvirt would be better off if it could auto-generate a client API for QEMU too, instead of writing it by hand from a human reading the QAPI
With regards,
Daniel
Cheers,
Victor
Quoted 33 lines · Victor Toso
Hi,
Quoted 30 lines · Markus Armbruster
Quoted 16 lines · Daniel P. Berrangé
Caller
block_resize(device="dev0", size=1*GiB) block_resize(node_name="devnode0", size=1*GiB)In golang definition
type BlockResizeArguments struct { Device string NodeName string Size int }Caller choice of
Quoted 9 lines
cmd := &BlockResizeCommand{ Device: "dev0", Size: 1 * GiB, } cmd := &BlockResizeCommand{ NodeName: "devnode0", Size: 1 * GiB, }Note that the Go bindings you sketched effectively use (poor man's) keyword arguments.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Neither case can easily prevent passing Device and NodeName at same time.
That defect lies at the schema's feet.
Right. The schema does not provide any metadata to explicit say that only @device or @node-name should be used, correct?
Correct.
The existing means to express either / or are tagged unions and alternates.
Tagged unions require an explicit tag member.
Alternates select the alternative by the type of the value.
We don't have anything that selects by member name.
Quoted 3 lines
This would be important to differentiate of a simple 'adding a new optional argument' plus 'making this other argument optional'.
We also don't have means to express "this integer must be a power of two", or "this string must name a block backend", or any number of semantic constraints.
We have to draw the line somewhere. Schema language complexity needs to earn its keep.
Quoted 53 lines
Quoted 53 lines
Quoted 14 lines
Quoted 2 lines
* At some future date, the old way gets deprecated: argument @device acquires feature @deprecated.
Ok, no change needed to the APIs in either case. Possibly have code emit a warning if a deprecated field is set.
Quoted 2 lines
* Still later, the old way gets removed: @device is deleted, and @node-name becomes mandatory.
Again no change needed to APIs, but QEMU will throw back an error if the wrong one is used.
Quoted 2 lines
What is the proper version-spanning interface?
Quoted 8 lines
Quoted 3 lines
I figure it's both arguments optional, must specify the right one for the version of QEMU actually in use. This spans versions, but it fails to abstract from them.
Yep, I think that's inevitable in this scenario. THe plus side is that apps that want to span versions can do so. The downside is that apps that don't want smarts to span version, may loose compile time warnings about use of the now deleted field.
The version-spanning interface will arguably be a bad interface for any version.
Quoted 26 lines
I suggested the code generator have an option to say what level of compat to use for generated code, so that apps can request an API without compat, which will result in compile errors. This though assumes every consumer app is embedding their own generated copy of the code. Not neccessarily desirable.
At the C level you can play games with
__deprecated__to get compile time warnings in some cases.#define
GLIB_VERSION_MIN_REQUIREDGLIB_VERSION_2_56causes QEMU to get compile time warnings (or errors) if it attempts to use a API feature deprecated in 2.56, even if the API exists in the header & library.
Quoted 6 lines
Note that it's not enough to replace "delete member" by "mark member deleted in <version>". You also have to keep full history for "is it optional". And for types, because those can evolve compatibly, too, e.g. from struct to flat union, or from string to alternate of string and something else. What is the proper version-spanning interface in all the possible cases?
I've not thought through all possible scenarios, but there may end up being restrictions, such that changes that were previously possible may have to be forbidden.
Quoted 8 lines
Quoted 3 lines
"There may be restrictions" is not exactly a confidence-inspring design assumption. We need a reasonably dependable idea on what exactly we're intending to sacrifice.
I can't help much here but I guess we can evolve QAPI schema as we move forward. Adding metadata that helps document changes to the benefit of giving code generators tools to provide a way to work with those QAPI changes seems desirable, no?
QMP comes with a certain compatibility promise. Keeping the promise has been expensive, but that's okay; a weaker promise would have been differently expensive. It's a compromise that has emerged over a long time.
I fail to see why QEMU-provided QMP bindings should come with a stronger promise than QEMU-provided QMP.
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
One example, in the past we could do deprecate a field 'foo', then delete 'foo' and then some time re-introduce 'foo' with a completely different type. That would not be possible if we wanted to maintain compat, but in this example that's probably a good thing, as it'll be super confusing to have the same field name change type like that over time. Easier to just use a different name.
So the question to me is not whether all our previous changes are still possible, but whether enough of the typwes of change are possible, such that we can cope with the ongoing maint in a reasonable way. I don't think we've explored the possibility enough to say one way or the other.
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Apps that wish to have version compat, would of course need to write their code to be aware of which fields they need to seend for which QEMU version.
At which point we're reinventing libvirt.
IMHO, at this moment, qapi-go is targeting communicating with QEMU and handling multiple QEMU versions seems reasonable to me.
It's targeting communicating in *QMP*. QMP is designed to support communicating with a range of QEMU versions. Full compatibility is promised for a narrow range. Outside that range, graceful degradation.
*If* you want to widen the full compatibility range, do it in *QMP*. Or do it on top of QEMU, e.g. in libvirt.
Quoted 2 lines
Perhaps libvirt can use qapi-go in the future or other generated interface. That would be cool.
"Would be cool" and a dollar buys you a cup of bad coffee.
Is it a good use of our limited resources?
How much will it delay delivery of Go bindings compared to less ambitious version?
Quoted 24 lines
Quoted 15 lines
Quoted 7 lines
The premise of the code generators is that there are applications that want to consume QEMU that are not libvirt. With this line of reasoning we could easily say that all such applications should just use libvirt and then we don't need to provide any of these code generators. The fact that we're considering these code generators though, says that we're accepting there are valid use cases that don't want to use libvirt for whatever reasons.
Can't resist any longer: "What has libvirt ever done for us?"
https://www.youtube.com/watch?v=Qc7HmhrgTuQQuoted 3 lines
It is
reasonable that some of those applications may wish to target a wide range of QEMU versions, just like libvirt does.Quoted 8 lines
At which point they're comitting to reinventing the relevant parts of libvirt.
I'd expect marshalling and umarshalling QMP to be among the smaller sub-problems then. It may look relatively large at first, because it's among the first ones to be solved. More so when you hand-wave away the more interesting ones of *abstraction* until they bite you in the posterior.
Quoted 8 lines
I might have missed it but I don't see unsolvable problems.
1) We decide if we want a Golang interface that can communicate
with multiple QEMU versions or not;2) We discuss how that Golang interface would look like;
3) We discuss what is needed in QEMU/QAPI to achieve (2)
4) We work on QEMU/QAPI to address (3)
5) We move on with qapi-go proposal
This reinvents significant parts of libvirt in QEMU.
Quoted 3 lines
I see only benefits with this project, plus the fact we already have Golang projects doing their own code to communicate with QEMU and I do believe we will make their lives easier.
Perhaps my idea of practical Go usage is hopelessly naive, but here goes anyway.
Go project vendors current Go bindings (because they vendor everything). Works with current QEMU, plus a range of past and future QEMUs.
Eventually something breaks with a then current QEMU. Go project updates vendored Go bindings to current. Compiler points out everything they need fixing (this assumes the type system is sufficiently expressive, and bindings use it properly).
Good enough for these Golang projects?
Or asked in a different way: what exactly are the compatibility requirements of these projects?
Please keep it to requirements *actual* projects have. Providing for additional, non-trivial requirements hypothetical projects might have is, in my reasoned opinion, folly we cannot afford. YAGNI until demonstrated otherwise.
If different projects have different requirements, then QEMU having to provide for all of them does not follow. Nothing absolves us from weighing benefits vs. costs and risks.
Quoted 14 lines · Markus Armbruster
Quoted 2 lines · Victor Toso
IMHO, at this moment, qapi-go is targeting communicating with QEMU and handling multiple QEMU versions seems reasonable to me.
It's targeting communicating in *QMP*. QMP is designed to support communicating with a range of QEMU versions. Full compatibility is promised for a narrow range. Outside that range, graceful degradation.
*If* you want to widen the full compatibility range, do it in *QMP*. Or do it on top of QEMU, e.g. in libvirt.
Quoted 2 lines
Perhaps libvirt can use qapi-go in the future or other generated interface. That would be cool.
Quoted 6 lines
"Would be cool" and a dollar buys you a cup of bad coffee.
Is it a good use of our limited resources?
How much will it delay delivery of Go bindings compared to less ambitious version?
Yeah, this thread has basically branched to cover three topics:
1. what an MVP Go interface for QMP should look like;
2. how to make sure said interface uses pretty names;
3. how to make it work across multiple QEMU versions.
All of these are important in the long run, but as far as I'm concerned only 1. is an actual blocker to making progress.
If we get to the point where we can generate a reasonably complete and well-typed Go interface that can be used to communicate with a single version of QEMU, we should just plaster EXPERIMENTAL all over it and get it merged.
Basically get the MVP done and then iterate over it in-tree rather than trying to get everything perfect from the start.
Sounds reasonable?
Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
Quoted 36 lines · Andrea Bolognani
Quoted 14 lines · Markus Armbruster
Quoted 2 lines · Victor Toso
IMHO, at this moment, qapi-go is targeting communicating with QEMU and handling multiple QEMU versions seems reasonable to me.
It's targeting communicating in *QMP*. QMP is designed to support communicating with a range of QEMU versions. Full compatibility is promised for a narrow range. Outside that range, graceful degradation.
*If* you want to widen the full compatibility range, do it in *QMP*. Or do it on top of QEMU, e.g. in libvirt.
Quoted 2 lines
Perhaps libvirt can use qapi-go in the future or other generated interface. That would be cool.
Quoted 6 lines
"Would be cool" and a dollar buys you a cup of bad coffee.
Is it a good use of our limited resources?
How much will it delay delivery of Go bindings compared to less ambitious version?
Yeah, this thread has basically branched to cover three topics:
1. what an MVP Go interface for QMP should look like;
2. how to make sure said interface uses pretty names;
3. how to make it work across multiple QEMU versions.All of these are important in the long run, but as far as I'm concerned only 1. is an actual blocker to making progress.
If we get to the point where we can generate a reasonably complete and well-typed Go interface that can be used to communicate with a single version of QEMU, we should just plaster EXPERIMENTAL all over it and get it merged.
Quoted 4 lines
Basically get the MVP done and then iterate over it in-tree rather than trying to get everything perfect from the start.
Sounds reasonable?
Yes, with an undogmatic interpretation of "minimally viable". Doing more should be okay when it doesn't complicate things outside the Go-generating backend.
Exploring how to generate Go bindings that make good use of static typing will be interesting enough. Aiming for wide compatibility in addition risks delay and/or failure. Exploding heads, too.
Hi,
Quoted 22 lines · Andrea Bolognani
Quoted 14 lines · Markus Armbruster
Quoted 2 lines · Victor Toso
IMHO, at this moment, qapi-go is targeting communicating with QEMU and handling multiple QEMU versions seems reasonable to me.
It's targeting communicating in *QMP*. QMP is designed to support communicating with a range of QEMU versions. Full compatibility is promised for a narrow range. Outside that range, graceful degradation.
*If* you want to widen the full compatibility range, do it in *QMP*. Or do it on top of QEMU, e.g. in libvirt.
Quoted 2 lines
Perhaps libvirt can use qapi-go in the future or other generated interface. That would be cool.
Quoted 6 lines
"Would be cool" and a dollar buys you a cup of bad coffee.
Is it a good use of our limited resources?
How much will it delay delivery of Go bindings compared to less ambitious version?
Quoted 8 lines
Yeah, this thread has basically branched to cover three topics:
1. what an MVP Go interface for QMP should look like;
2. how to make sure said interface uses pretty names;
3. how to make it work across multiple QEMU versions.All of these are important in the long run, but as far as I'm concerned only 1. is an actual blocker to making progress.
I agree although (1) and (3) are holding hands a bit.
Quoted 9 lines
If we get to the point where we can generate a reasonably complete and well-typed Go interface that can be used to communicate with a single version of QEMU, we should just plaster EXPERIMENTAL all over it and get it merged.
Basically get the MVP done and then iterate over it in-tree rather than trying to get everything perfect from the start.
Sounds reasonable?
Yep. The whole discussion has been great as to clarify limitations and possible goals but not aiming to get it perfect all at once seems reasonable.
Cheers,
Victor
Quoted 9 lines · Victor Toso
* Status
There are a few rough edges to work on but this is usable. The major thing I forgot to add is handling Error from Commands. It'll be the first thing I'll work on next week.
If you want to start using this Today you can fetch it in at
Looking at this my main concern is that there is way too much use of the 'Any' type. The valid Golang types that can be used with any of the 'Any' fields are merely expresssed as comments in the code. I think this needs changing so that the Golang types are directly expressed in code.
I think there are perhaps only 2-3 cases where the 'Any' type is genuinely neccessary due to the QAPI schema having an unbounded set of possible types - SchemaInfoObjectMember, ObjectPropertyInfo and QomSetCommand
With regards,
Daniel
Signature · 3 lines
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|