Skip to contentSkip to conversation

qapi: add generator for Golang interface

RFC PATCH Superseded Victor Toso 5 participants 71 messages Apr 1, 2022 – Jun 1, 2022 Arm/Amlogic Meson SoC support

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover qapi: add generator for Golang interface

Hi,

Happy 1st April. Not a joke :) /* ugh, took me too long to send */

This series is about adding a generator in scripts/qapi to produce Go data structures that can be used to communicate with QEMU over QMP.

* Why Go?

There are quite a few Go projects that interact with QEMU over QMP and they endup using a mix of different libraries with their own code.

** Which projects?

The ones I've found so far:

- podman machine
https://github.com/containers/podman/tree/main/pkg/machine/qemu

- kata-containers (govmm)
https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm

- lxd
https://github.com/lxc/lxd/tree/master/lxd/instance/drivers

- kubevirt (plain json strings)
https://github.com/kubevirt/kubevirt

(let me know if you know others)

* But Why?

I'm particularly interested in 3 out of 4 of the projects above and only Kubevirt uses libvirt to handle QEMU. That means that every QEMU releases where a QMP command, event or other data struct is added, removed or changed, those projects need to check what changed in QEMU and then address those changes in their projects, if needed.

The idea behind generating Go data structures is that we can keep a Go module which can have releases that follow QEMU releases.

The project that uses this Go module, only need to bump the module version and it shall receive all the changes in their own vendored code base.

* Status

There are a few rough edges to work on but this is usable. The major thing I forgot to add is handling Error from Commands. It'll be the first thing I'll work on next week.

If you want to start using this Today you can fetch it in at

https://gitlab.com/victortoso/qapi-go/

There are quite a few tests that I took from the examples in the qapi schema. Coverage using go's cover tool is giving 28.6% of statements

I've uploaded the a static generated godoc output of the above Go module here:

https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/

* License

While the generator (golang.py in this series) is GPL v2, the generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.

* Disclaimer to reviewers

This is my first serious python project so there'll be lots of suggetions that I'll be happy to take and learn from.

Thanks for taking a look, let me know if you have questions, ideas or suggestions.

Cheers,
Victor

Victor Toso (8):
qapi: golang: Generate qapi's enum types in Go
qapi: golang: Generate qapi's alternate types in Go
qapi: golang: Generate qapi's struct types in Go
qapi: golang: Generate qapi's union types in Go
qapi: golang: Generate qapi's event types in Go
qapi: golang: Generate qapi's command types in Go
qapi: golang: Add CommandResult type to Go
qapi: golang: document skip function visit_array_types

 qapi/meson.build       |   1 +
 scripts/qapi/golang.py | 727 +++++++++++++++++++++++++++++++++++++++++
 scripts/qapi/main.py   |   2 +
 3 files changed, 730 insertions(+)
 create mode 100644 scripts/qapi/golang.py
Signature · 1 line
2.35.1
#1 lore raw

RFC PATCH 1/8 qapi: golang: Generate qapi's enum types in Go

This patch handles QAPI enum types and generates its equivalent in Go.

The highlights of this implementation are:

1. For each QAPI enum, we will define an int32 type in Go to be the assigned type of this specific enum

2. While in the Go codebase we can use the generated enum values, the specification requires that, on the wire, the enumeration type's value to be represented by its string name.

For this reason, each Go type that represent's a QAPI enum will be implementing the Marshaler[0] and Unmarshaler[1] interfaces to seamless handle QMP's string to Go int32 value and vice-versa.

3. Naming: CamelCase will be used in any identifier that we want to export [2], which is everything in this patch.

[0] https://pkg.go.dev/encoding/json#Marshaler
[1] https://pkg.go.dev/encoding/json#Unmarshaler
[2] https://go.dev/ref/spec#Exported_identifiers

---
 qapi/meson.build       |   1 +
 scripts/qapi/golang.py | 225 +++++++++++++++++++++++++++++++++++++++++
 scripts/qapi/main.py   |   2 +
 3 files changed, 228 insertions(+)
 create mode 100644 scripts/qapi/golang.py
qapi/meson.build +1 -0
index 656ef0e039..0951692332 100644
@@ -90,6 +90,7 @@ qapi_nonmodule_outputs = [
   'qapi-init-commands.h', 'qapi-init-commands.c',
   'qapi-events.h', 'qapi-events.c',
   'qapi-emit-events.c', 'qapi-emit-events.h',
+  'qapibara.go',
 ]
 
 # First build all sources
scripts/qapi/golang.py new file +225 -0

225 lines not on this page · raw message

scripts/qapi/main.py +2 -0
index fc216a53d3..661fb1e091 100644
@@ -15,6 +15,7 @@
 from .common import must_match
 from .error import QAPIError
 from .events import gen_events
+from .golang import gen_golang
 from .introspect import gen_introspect
 from .schema import QAPISchema
 from .types import gen_types
@@ -54,6 +55,7 @@ def generate(schema_file: str,
     gen_events(schema, output_dir, prefix)
     gen_introspect(schema, output_dir, prefix, unmask)
 
+    gen_golang(schema, output_dir, prefix)
 
 def main() -> int:
     """
Signature · 1 line
2.35.1
#2 lore raw

Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go

Quoted 6 lines · Victor Toso

This patch handles QAPI enum types and generates its equivalent in Go.

The highlights of this implementation are:

1. For each QAPI enum, we will define an int32 type in Go to be the assigned type of this specific enum

I think there's a potential case to be made for considering representation of enums as strings in Golang, as it more gracefully degrades.

Consider the 'SHUTDOWN' event in QEMU, which has a 'reason' field.

This implementation has

type ShutdownCause int32

const (
ShutdownCauseNone ShutdownCause = iota
ShutdownCauseHostError // An error prevents further use of guest
ShutdownCauseHostQmpQuit // Reaction to the QMP command 'quit'
ShutdownCauseHostQmpSystemReset // Reaction to the QMP command 'system_reset'
ShutdownCauseHostSignal // Reaction to a signal, such as SIGINT
ShutdownCauseHostUi // Reaction to a UI event, like window close
ShutdownCauseGuestShutdown // Guest shutdown/suspend request, via ACPI or other hardware-specific means
ShutdownCauseGuestReset // Guest reset request, and command line turns that into a shutdown
ShutdownCauseGuestPanic // Guest panicked, and command line turns that into a shutdown
ShutdownCauseSubsystemReset // Partial guest reset that does not trigger QMP events and ignores --no-reboot. This is useful for sanitizing hypercalls on s390 that are used during kexec/kdump/boot
)

and

type ShutdownEvent struct {
Guest bool json:"guest" // If true, the shutdown was triggered by a guest request (such as a guest-initiated ACPI shutdown request or other hardware-specific action) rather than a host request (such as sending qemu a SIGINT). (since 2.10)
Reason ShutdownCause json:"reason" // The @ShutdownCause which resulted in the SHUTDOWN. (since 4.0)
}

Consider that the application is compiled against the QAPI generated API from QEMU 7.0. The application is believed to run happily against 7.1, because app doesn't need any of the functionality added in 7.1 QAPI. But consider that QEMU 7.1 had introduced a new shutdown reason value.

The application may want to know that the guest has shutdown, but does NOT care about the reason for the shutdown.

Since the ShutdownReason is implemented as an int though, the unmarshalling for the reason field is going to fail.

If the enums were just represented as strings, then we can gracefully accept any new enum value that arrives in future. The application can thus also still log the shutdown reason string, even though it was not a value explicitly known to the generated API.

Quoted 11 lines

2. While in the Go codebase we can use the generated enum values, the specification requires that, on the wire, the enumeration type's value to be represented by its string name.

For this reason, each Go type that represent's a QAPI enum will be implementing the Marshaler[0] and Unmarshaler[1] interfaces to seamless handle QMP's string to Go int32 value and vice-versa.

3. Naming: CamelCase will be used in any identifier that we want to export [2], which is everything in this patch.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#3 lore raw

Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go

Hi,

Quoted 44 lines · Daniel P. Berrangé
Quoted 6 lines · Victor Toso

This patch handles QAPI enum types and generates its equivalent in Go.

The highlights of this implementation are:

1. For each QAPI enum, we will define an int32 type in Go to be the assigned type of this specific enum

I think there's a potential case to be made for considering representation of enums as strings in Golang, as it more gracefully degrades.

Consider the 'SHUTDOWN' event in QEMU, which has a 'reason' field.

This implementation has

type ShutdownCause int32

const (
ShutdownCauseNone ShutdownCause = iota
ShutdownCauseHostError // An error prevents further use of guest
ShutdownCauseHostQmpQuit // Reaction to the QMP command 'quit'
ShutdownCauseHostQmpSystemReset // Reaction to the QMP command 'system_reset'
ShutdownCauseHostSignal // Reaction to a signal, such as SIGINT
ShutdownCauseHostUi // Reaction to a UI event, like window close
ShutdownCauseGuestShutdown // Guest shutdown/suspend request, via ACPI or other hardware-specific means
ShutdownCauseGuestReset // Guest reset request, and command line turns that into a shutdown
ShutdownCauseGuestPanic // Guest panicked, and command line turns that into a shutdown
ShutdownCauseSubsystemReset // Partial guest reset that does not trigger QMP events and ignores --no-reboot. This is useful for sanitizing hypercalls on s390 that are used during kexec/kdump/boot
)

and

type ShutdownEvent struct {
Guest bool json:"guest" // If true, the shutdown was triggered by a guest request (such as a guest-initiated ACPI shutdown request or other hardware-specific action) rather than a host request (such as sending qemu a SIGINT). (since 2.10)
Reason ShutdownCause json:"reason" // The @ShutdownCause which resulted in the SHUTDOWN. (since 4.0)
}

Consider that the application is compiled against the QAPI generated API from QEMU 7.0. The application is believed to run happily against 7.1, because app doesn't need any of the functionality added in 7.1 QAPI. But consider that QEMU 7.1 had introduced a new shutdown reason value.

Quoted 5 lines

The application may want to know that the guest has shutdown, but does NOT care about the reason for the shutdown.

Since the ShutdownReason is implemented as an int though, the unmarshalling for the reason field is going to fail.

Marshalling does error if you try to convert an int that is not in the range of the enum type.

Unmarshalling should not error in this case, but the field ends up not being set which defaults to 0 (in this case, ShutdownCauseNone). That could mislead the *actual* reason but not without a warning which is expected in this case, imho.

(I know is not an actual warning, just a Println, but it'll be fixed in the next iteration)

  | func (s *ShutdownCause) UnmarshalJSON(data []byte) error {
  |     var name string
  |
  |     if err := json.Unmarshal(data, &name); err != nil {
  |         return err
  |     }
  |
  |     switch name {
  |     case "none":
  |         (*s) = ShutdownCauseNone
  |     case "host-error":
  |         (*s) = ShutdownCauseHostError
  |     case "host-qmp-quit":
  |         (*s) = ShutdownCauseHostQmpQuit
  |     case "host-qmp-system-reset":
  |         (*s) = ShutdownCauseHostQmpSystemReset
  |     case "host-signal":
  |         (*s) = ShutdownCauseHostSignal
  |     case "host-ui":
  |         (*s) = ShutdownCauseHostUi
  |     case "guest-shutdown":
  |         (*s) = ShutdownCauseGuestShutdown
  |     case "guest-reset":
  |         (*s) = ShutdownCauseGuestReset
  |     case "guest-panic":
  |         (*s) = ShutdownCauseGuestPanic
  |     case "subsystem-reset":
  |         (*s) = ShutdownCauseSubsystemReset
  |     default:
  |         fmt.Println("Failed to decode ShutdownCause", *s)
  |     }
  |     return nil
  | }
Quoted 5 lines

If the enums were just represented as strings, then we can gracefully accept any new enum value that arrives in future. The application can thus also still log the shutdown reason string, even though it was not a value explicitly known to the generated API.

As a string, the warning should still exist and default value of "" or nil for ptr would apply. IMHO, between string + warning and int + warning, I'd still go with int here.

That's a design decision to be made. All in all, I don't know much about the changes in QEMU/QAPI per version to take this decision, so I'll rely on you and the list for this, not just for enums but for the other types too.

Quoted 10 lines
Quoted 10 lines

2. While in the Go codebase we can use the generated enum values, the specification requires that, on the wire, the enumeration type's value to be represented by its string name.

For this reason, each Go type that represent's a QAPI enum will be implementing the Marshaler[0] and Unmarshaler[1] interfaces to seamless handle QMP's string to Go int32 value and vice-versa.

3. Naming: CamelCase will be used in any identifier that we want to export [2], which is everything in this patch.

Cheers,
Victor

#4 lore raw

Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go

Quoted 56 lines · Victor Toso

Hi,

Quoted 44 lines · Daniel P. Berrangé
Quoted 6 lines · Victor Toso

This patch handles QAPI enum types and generates its equivalent in Go.

The highlights of this implementation are:

1. For each QAPI enum, we will define an int32 type in Go to be the assigned type of this specific enum

I think there's a potential case to be made for considering representation of enums as strings in Golang, as it more gracefully degrades.

Consider the 'SHUTDOWN' event in QEMU, which has a 'reason' field.

This implementation has

type ShutdownCause int32

const (
ShutdownCauseNone ShutdownCause = iota
ShutdownCauseHostError // An error prevents further use of guest
ShutdownCauseHostQmpQuit // Reaction to the QMP command 'quit'
ShutdownCauseHostQmpSystemReset // Reaction to the QMP command 'system_reset'
ShutdownCauseHostSignal // Reaction to a signal, such as SIGINT
ShutdownCauseHostUi // Reaction to a UI event, like window close
ShutdownCauseGuestShutdown // Guest shutdown/suspend request, via ACPI or other hardware-specific means
ShutdownCauseGuestReset // Guest reset request, and command line turns that into a shutdown
ShutdownCauseGuestPanic // Guest panicked, and command line turns that into a shutdown
ShutdownCauseSubsystemReset // Partial guest reset that does not trigger QMP events and ignores --no-reboot. This is useful for sanitizing hypercalls on s390 that are used during kexec/kdump/boot
)

and

type ShutdownEvent struct {
Guest bool json:"guest" // If true, the shutdown was triggered by a guest request (such as a guest-initiated ACPI shutdown request or other hardware-specific action) rather than a host request (such as sending qemu a SIGINT). (since 2.10)
Reason ShutdownCause json:"reason" // The @ShutdownCause which resulted in the SHUTDOWN. (since 4.0)
}

Consider that the application is compiled against the QAPI generated API from QEMU 7.0. The application is believed to run happily against 7.1, because app doesn't need any of the functionality added in 7.1 QAPI. But consider that QEMU 7.1 had introduced a new shutdown reason value.

Quoted 5 lines

The application may want to know that the guest has shutdown, but does NOT care about the reason for the shutdown.

Since the ShutdownReason is implemented as an int though, the unmarshalling for the reason field is going to fail.

Marshalling does error if you try to convert an int that is not in the range of the enum type.

Quoted 7 lines

Unmarshalling should not error in this case, but the field ends up not being set which defaults to 0 (in this case, ShutdownCauseNone). That could mislead the *actual* reason but not without a warning which is expected in this case, imho.

(I know is not an actual warning, just a Println, but it'll be fixed in the next iteration)

I don't thinnk we should be emitting warnings/printlns at all in this case though. The app should be able to consume the enum without needing a warning. If the app wants to validate against a specific set of constants, it can decide to emit a warning if there was a case it can't handle. We shouldn't emit warnings/errors in the unmarshalling step though,as it isn't the right place to decide on such policy.

Quoted 41 lines
  | func (s *ShutdownCause) UnmarshalJSON(data []byte) error {
  |     var name string
  |
  |     if err := json.Unmarshal(data, &name); err != nil {
  |         return err
  |     }
  |
  |     switch name {
  |     case "none":
  |         (*s) = ShutdownCauseNone
  |     case "host-error":
  |         (*s) = ShutdownCauseHostError
  |     case "host-qmp-quit":
  |         (*s) = ShutdownCauseHostQmpQuit
  |     case "host-qmp-system-reset":
  |         (*s) = ShutdownCauseHostQmpSystemReset
  |     case "host-signal":
  |         (*s) = ShutdownCauseHostSignal
  |     case "host-ui":
  |         (*s) = ShutdownCauseHostUi
  |     case "guest-shutdown":
  |         (*s) = ShutdownCauseGuestShutdown
  |     case "guest-reset":
  |         (*s) = ShutdownCauseGuestReset
  |     case "guest-panic":
  |         (*s) = ShutdownCauseGuestPanic
  |     case "subsystem-reset":
  |         (*s) = ShutdownCauseSubsystemReset
  |     default:
  |         fmt.Println("Failed to decode ShutdownCause", *s)
  |     }
  |     return nil
  | }
Quoted 5 lines

If the enums were just represented as strings, then we can gracefully accept any new enum value that arrives in future. The application can thus also still log the shutdown reason string, even though it was not a value explicitly known to the generated API.

Quoted 8 lines

As a string, the warning should still exist and default value of "" or nil for ptr would apply. IMHO, between string + warning and int + warning, I'd still go with int here.

That's a design decision to be made. All in all, I don't know much about the changes in QEMU/QAPI per version to take this decision, so I'll rely on you and the list for this, not just for enums but for the other types too.

Essentially every release of QEMU will have QAPI changes. Most of the time these are append-only changes. ie a new struct, new command, new field, new enum value. Sometimes there will be removals due to deprecation, though note my other reply saying that we really ought to stop doing removals from the schema, and instead just annotate when a field stops being used.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#5 lore raw

Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go

Hi,

Quoted 11 lines · Daniel P. Berrangé
Quoted 10 lines

Marshalling does error if you try to convert an int that is not in the range of the enum type.

Unmarshalling should not error in this case, but the field ends up not being set which defaults to 0 (in this case, ShutdownCauseNone). That could mislead the *actual* reason but not without a warning which is expected in this case, imho.

(I know is not an actual warning, just a Println, but it'll be fixed in the next iteration)

Quoted 7 lines

I don't thinnk we should be emitting warnings/printlns at all in this case though. The app should be able to consume the enum without needing a warning. If the app wants to validate against a specific set of constants, it can decide to emit a warning if there was a case it can't handle. We shouldn't emit warnings/errors in the unmarshalling step though,as it isn't the right place to decide on such policy.

I think it is useful to know that, a binary compiled to qapi-go 7.0 but running with qemu 7.1 would have some mismatches. It could help detect issues (e.g: Why my program doesn't know/show the reason for shutdown?).

So, some sort of --verbose option for this level should exist.

Quoted 49 lines
Quoted 40 lines
  | func (s *ShutdownCause) UnmarshalJSON(data []byte) error {
  |     var name string
  |
  |     if err := json.Unmarshal(data, &name); err != nil {
  |         return err
  |     }
  |
  |     switch name {
  |     case "none":
  |         (*s) = ShutdownCauseNone
  |     case "host-error":
  |         (*s) = ShutdownCauseHostError
  |     case "host-qmp-quit":
  |         (*s) = ShutdownCauseHostQmpQuit
  |     case "host-qmp-system-reset":
  |         (*s) = ShutdownCauseHostQmpSystemReset
  |     case "host-signal":
  |         (*s) = ShutdownCauseHostSignal
  |     case "host-ui":
  |         (*s) = ShutdownCauseHostUi
  |     case "guest-shutdown":
  |         (*s) = ShutdownCauseGuestShutdown
  |     case "guest-reset":
  |         (*s) = ShutdownCauseGuestReset
  |     case "guest-panic":
  |         (*s) = ShutdownCauseGuestPanic
  |     case "subsystem-reset":
  |         (*s) = ShutdownCauseSubsystemReset
  |     default:
  |         fmt.Println("Failed to decode ShutdownCause", *s)
  |     }
  |     return nil
  | }
Quoted 5 lines

If the enums were just represented as strings, then we can gracefully accept any new enum value that arrives in future. The application can thus also still log the shutdown reason string, even though it was not a value explicitly known to the generated API.

Quoted 8 lines

As a string, the warning should still exist and default value of "" or nil for ptr would apply. IMHO, between string + warning and int + warning, I'd still go with int here.

That's a design decision to be made. All in all, I don't know much about the changes in QEMU/QAPI per version to take this decision, so I'll rely on you and the list for this, not just for enums but for the other types too.

Quoted 6 lines

Essentially every release of QEMU will have QAPI changes. Most of the time these are append-only changes. ie a new struct, new command, new field, new enum value. Sometimes there will be removals due to deprecation, though note my other reply saying that we really ought to stop doing removals from the schema, and instead just annotate when a field stops being used.

Ok, thanks.

Cheers,
Victor

#6 lore raw

Re: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go

Quoted 22 lines · Victor Toso

Hi,

Quoted 11 lines · Daniel P. Berrangé
Quoted 10 lines

Marshalling does error if you try to convert an int that is not in the range of the enum type.

Unmarshalling should not error in this case, but the field ends up not being set which defaults to 0 (in this case, ShutdownCauseNone). That could mislead the *actual* reason but not without a warning which is expected in this case, imho.

(I know is not an actual warning, just a Println, but it'll be fixed in the next iteration)

Quoted 7 lines

I don't thinnk we should be emitting warnings/printlns at all in this case though. The app should be able to consume the enum without needing a warning. If the app wants to validate against a specific set of constants, it can decide to emit a warning if there was a case it can't handle. We shouldn't emit warnings/errors in the unmarshalling step though,as it isn't the right place to decide on such policy.

Quoted 4 lines

I think it is useful to know that, a binary compiled to qapi-go 7.0 but running with qemu 7.1 would have some mismatches. It could help detect issues (e.g: Why my program doesn't know/show the reason for shutdown?).

If apps are actually looking at the shutdown reasons, it is trivial for the app themselves to log any unexpected reasons.

They should in fact do that regardless, because if someone re-compiles the app against a new version of the QEMU Go API, there may be new ShutdownCause's defined that their codes does not handle. There's no compile time check from Go side that their code has handled all ShutdownCause values, since there's no struct enum type in Go. This again pushes me to towards the view that the enums should just remain as strings, the use of integers gives no functional benefit AFAICT, given the lack of a real enum type in Go.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#7 lore raw

RFC PATCH 2/8 qapi: golang: Generate qapi's alternate types in Go

In reply to Victor Toso

This patch handles QAPI alternate types and generates data structures in Go that handles it.

At this moment, there are 5 alternates in qemu/qapi, they are:
* BlockDirtyBitmapMergeSource
* Qcow2OverlapChecks
* BlockdevRef
* BlockdevRefOrNull
* StrOrNull

Alternate types are similar to Union but without a discriminator that can be used to identify the underlying value on the wire. It is needed to infer it. That can't be easily mapped in Go.

For each Alternate type, we will be using a Any type to hold the value. 'Any' is an alias type for interface{} (similar to void* in C).

Similarly to the Enum types (see previous commit), we will implement Marshaler and Unmarshaler interfaces for the Alternate types and in those MarshalJSON() and UnmarshalJSON() methods is where we are going to put the logic to read/set alternate's value.

Note that on UnmarshalJSON(), a helper function called StrictDecode() will be used. This function is the main logic to infer if a given JSON object fits in a given Go struct. Because we only have 5 alternate types, it is not hard to validate the unmarshaling logic but we might need to improve it in the future if Alternate with branches that have similar fields appear.

Examples:
* BlockdevRef

```go
    // Data to set in BlockdevOptions
    qcow2 := BlockdevOptionsQcow2{}
    // BlockdevRef using a string
    qcow2.File = BlockdevRef{Value: "/some/place/my-image"}
    opt := BlockdevOptions{}
    opt.Driver = BlockdevDriverQcow2
    opt.Value = qcow2

    b, _ := json.Marshal(data.s)
    // string(b) == `{"driver":"qcow2","file":"/some/place/my-image"}`
```
---
 scripts/qapi/golang.py | 157 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 155 insertions(+), 2 deletions(-)
scripts/qapi/golang.py +155 -2
index 070d4cbbae..8be31bd902 100644
@@ -31,7 +31,8 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
 
     def __init__(self, prefix: str):
         super().__init__()
-        self.target = {name: "" for name in ["enum"]}
+        self.target = {name: "" for name in ["alternate", "enum", "helper"]}
+        self.objects_seen = {}
         self.schema = None
         self._docmap = {}
         self.golang_package_name = "qapi"
@@ -43,6 +44,10 @@ def visit_begin(self, schema):
         for target in self.target:
             self.target[target] = f"package {self.golang_package_name}\n"
 
+        self.target["helper"] += f'''
+    // Alias for go version lower than 1.18
+    type Any = interface{{}}'''
+
         # Iterate once in schema.docs to map doc objects to its name
         for doc in schema.docs:
             if doc.symbol is None:
@@ -52,6 +57,22 @@ def visit_begin(self, schema):
     def visit_end(self):
         self.schema = None
 
+        self.target["helper"] += '''
+// Creates a decoder that errors on unknown Fields
+// Returns true if successfully decoded @from string @into type
+// Returns false without error is failed with "unknown field"
+// Returns false with error is a different error was found
+func StrictDecode(into interface{}, from []byte) error {
+	dec := json.NewDecoder(strings.NewReader(string(from)))
+	dec.DisallowUnknownFields()
+
+    if err := dec.Decode(into); err != nil {
+        return err
+    }
+	return nil
+}
+'''
+
     def visit_object_type(self: QAPISchemaGenGolangVisitor,
                           name: str,
                           info: Optional[QAPISourceInfo],
@@ -70,7 +91,123 @@ def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
                              features: List[QAPISchemaFeature],
                              variants: QAPISchemaVariants
                              ) -> None:
-        pass
+        assert name not in self.objects_seen
+        self.objects_seen[name] = True
+
+        # Alternate marshal logic
+        #
+        # To avoid programming errors by users of this generated Go module,
+        # we add a runtime check to error out in case the underlying Go type
+        # doesn't not match any of supported types of the Alternate type.
+        #
+        # Also, Golang's json Marshal will include as JSON's object, the
+        # wrapper we use to hold the Go struct (Value Any -> `Value: {...}`)
+        # This would not be an valid QMP message so we workaround it by
+        # calling RemoveValueObject function.
+        doc = self._docmap.get(name, None)
+        doc_struct, doc_fields = qapi_to_golang_struct_docs(doc)
+
+        members_doc = '''// Options are:'''
+        if_supported_types = ""
+        for var in variants.variants:
+            field_doc = doc_fields.get(var.name, "")
+            field_go_type = qapi_schema_type_to_go_type(var.type.name)
+            members_doc += f'''\n// * {var.name} ({field_go_type}):{field_doc[3:]}'''
+
+            if field_go_type == "nil":
+                field_go_type = "*string"
+
+            if_supported_types += f'''typestr != "{field_go_type}" &&\n\t\t'''
+
+        # Alternate unmarshal logic
+        #
+        # With Alternate types, we have to check the JSON data in order to
+        # identify what is the target Go type. So, this is different than an
+        # union which has an identifier that we can check first.
+        # StrictDecode function tries to match the given JSON data to a given
+        # Go type and it'll error in case it doesn´t fit, for instance, when
+        # there were members in the JSON data that had no equivalent in the
+        # target Go type.
+        #
+        # For this reason, the order is important.
+        #
+        # At this moment, the only field that must be checked first is JSON
+        # NULL, which is relevant to a few alternate types. In the future, we
+        # might need to improve the logic to be foolproof between target Go
+        # types that might have a common base (non existing Today).
+        check_type_str = '''
+    // Check for {name}
+    {{
+        var value {go_type}
+        if err := StrictDecode(&value, data); {error_check} {{
+            s.Value = {set_value}
+            return nil
+        }}
+    }}'''
+        reference_checks = ""
+        for var in variants.variants:
+            if var.type.name == "null":
+                # We use a pointer (by referece) to check for JSON's NULL
+                reference_checks += check_type_str.format(
+                        name = var.type.name,
+                        go_type = "*string",
+                        error_check = "err == nil && value == nil",
+                        set_value = "nil")
+                break;
+
+        value_checks = ""
+        for var in variants.variants:
+            if var.type.name != "null":
+                go_type = qapi_schema_type_to_go_type(var.type.name)
+                value_checks += check_type_str.format(
+                        name = var.type.name,
+                        go_type = go_type,
+                        error_check = "err == nil",
+                        set_value = "value")
+
+        unmarshal_checks = ""
+        if len(reference_checks) > 0 and len(value_checks) > 0:
+            unmarshal_checks = reference_checks[1:] + value_checks
+        else:
+            unmarshal_checks = reference_checks[1:] if len(reference_checks) > 0 else value_checks[1:]
+
+        self.target["alternate"] += f'''
+{doc_struct}
+type {name} struct {{
+{members_doc}
+    Value Any
+}}
+
+func (s {name}) MarshalJSON() ([]byte, error) {{
+    typestr := fmt.Sprintf("%T", s.Value)
+    typestr = typestr[strings.LastIndex(typestr, ".")+1:]
+
+    // Runtime check for supported types
+    if typestr != "<nil>" &&
+{if_supported_types[:-6]} {{
+        return nil, errors.New(fmt.Sprintf("Type is not supported: %s", typestr))
+    }}
+
+    b, err := json.Marshal(s.Value);
+    if err != nil {{
+        return nil, err
+    }}
+
+    return b, nil
+}}
+
+func (s *{name}) UnmarshalJSON(data []byte) error {{
+{unmarshal_checks}
+    // Check type to error out nicely
+    {{
+        var value Any
+        if err := json.Unmarshal(data, &value); err != nil {{
+            return err
+        }}
+        return errors.New(fmt.Sprintf("Unsupported type %T (value: %v)", value, value))
+    }}
+}}
+'''
 
     def visit_enum_type(self: QAPISchemaGenGolangVisitor,
                         name: str,
@@ -208,6 +345,22 @@ def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]):
 
     return main, fields
 
+def qapi_schema_type_to_go_type(type: str) -> str:
+    schema_types_to_go = {'str': 'string', 'null': 'nil', 'bool': 'bool',
+            'number': 'float64', 'size': 'uint64', 'int': 'int64', 'int8': 'int8',
+            'int16': 'int16', 'int32': 'int32', 'int64': 'int64', 'uint8': 'uint8',
+            'uint16': 'uint16', 'uint32': 'uint32', 'uint64': 'uint64',
+            'any': 'Any', 'QType': 'QType',
+    }
+
+    prefix = ""
+    if type.endswith("List"):
+        prefix = "[]"
+        type = type[:-4]
+
+    type = schema_types_to_go.get(type, type)
+    return prefix + type
+
 def qapi_to_field_name_enum(name: str) -> str:
     return name.title().replace("-", "")
 
Signature · 1 line
2.35.1
#8 lore raw

Re: [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate types in Go

Quoted 10 lines · Victor Toso

This patch handles QAPI alternate types and generates data structures in Go that handles it.

At this moment, there are 5 alternates in qemu/qapi, they are:
* BlockDirtyBitmapMergeSource
* Qcow2OverlapChecks
* BlockdevRef
* BlockdevRefOrNull
* StrOrNull

Quoted 3 lines

Alternate types are similar to Union but without a discriminator that can be used to identify the underlying value on the wire. It is needed to infer it. That can't be easily mapped in Go.

I don't buy that. Given this example:

  type BlockdevRef struct {
        // Options are:
        // * definition (BlockdevOptions): defines a new block device inline
        // * reference (string): references the ID of an existing block device
        Value Any

}

What is the problem with having this Go struct:

type BlockdevRef struct {
Definition *BlockdevOptions
Reference *string
}

when deserializing from JSON, we know exactly which one of these two fields to populate. The programmer consuming this can look at which field is non-nil.

When serializing to JSON, we serialize which ever field is non-nil.

If both fields are non-nil that's a programmer bug. Either ignore it and only serialize the first non-nil field, or raise an error.

Quoted 31 lines

For each Alternate type, we will be using a Any type to hold the value. 'Any' is an alias type for interface{} (similar to void* in C).

Similarly to the Enum types (see previous commit), we will implement Marshaler and Unmarshaler interfaces for the Alternate types and in those MarshalJSON() and UnmarshalJSON() methods is where we are going to put the logic to read/set alternate's value.

Note that on UnmarshalJSON(), a helper function called StrictDecode() will be used. This function is the main logic to infer if a given JSON object fits in a given Go struct. Because we only have 5 alternate types, it is not hard to validate the unmarshaling logic but we might need to improve it in the future if Alternate with branches that have similar fields appear.

Examples:
* BlockdevRef

```go
    // Data to set in BlockdevOptions
    qcow2 := BlockdevOptionsQcow2{}
    // BlockdevRef using a string
    qcow2.File = BlockdevRef{Value: "/some/place/my-image"}
    opt := BlockdevOptions{}
    opt.Driver = BlockdevDriverQcow2
    opt.Value = qcow2

    b, _ := json.Marshal(data.s)
    // string(b) == `{"driver":"qcow2","file":"/some/place/my-image"}`
```
Quoted 4 lines
---
 scripts/qapi/golang.py | 157 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 155 insertions(+), 2 deletions(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#9 lore raw

Re: [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate types in Go

Hi,

Quoted 23 lines · Daniel P. Berrangé
Quoted 10 lines · Victor Toso

This patch handles QAPI alternate types and generates data structures in Go that handles it.

At this moment, there are 5 alternates in qemu/qapi, they are:
* BlockDirtyBitmapMergeSource
* Qcow2OverlapChecks
* BlockdevRef
* BlockdevRefOrNull
* StrOrNull

Quoted 4 lines

Alternate types are similar to Union but without a discriminator that can be used to identify the underlying value on the wire. It is needed to infer it. That can't be easily mapped in Go.

I don't buy that. Given this example:

  type BlockdevRef struct {
        // Options are:
        // * definition (BlockdevOptions): defines a new block device inline
        // * reference (string): references the ID of an existing block device
        Value Any
Quoted 8 lines

}

What is the problem with having this Go struct:

type BlockdevRef struct {
Definition *BlockdevOptions
Reference *string
}

... this is better.

Quoted 10 lines

when deserializing from JSON, we know exactly which one of these two fields to populate. The programmer consuming this can look at which field is non-nil.

When serializing to JSON, we serialize which ever field is non-nil.

If both fields are non-nil that's a programmer bug. Either ignore it and only serialize the first non-nil field, or raise an error.

It would be a programmer bug if they set a Value of a type not allowed by Type's spec, but it would be a *runtime* error. Your suggestion is more type safe.

Thanks.

Quoted 44 lines
Quoted 32 lines

For each Alternate type, we will be using a Any type to hold the value. 'Any' is an alias type for interface{} (similar to void* in C).

Similarly to the Enum types (see previous commit), we will implement Marshaler and Unmarshaler interfaces for the Alternate types and in those MarshalJSON() and UnmarshalJSON() methods is where we are going to put the logic to read/set alternate's value.

Note that on UnmarshalJSON(), a helper function called StrictDecode() will be used. This function is the main logic to infer if a given JSON object fits in a given Go struct. Because we only have 5 alternate types, it is not hard to validate the unmarshaling logic but we might need to improve it in the future if Alternate with branches that have similar fields appear.

Examples:
* BlockdevRef

```go
    // Data to set in BlockdevOptions
    qcow2 := BlockdevOptionsQcow2{}
    // BlockdevRef using a string
    qcow2.File = BlockdevRef{Value: "/some/place/my-image"}
    opt := BlockdevOptions{}
    opt.Driver = BlockdevDriverQcow2
    opt.Value = qcow2

    b, _ := json.Marshal(data.s)
    // string(b) == `{"driver":"qcow2","file":"/some/place/my-image"}`
```
Quoted 4 lines
---
 scripts/qapi/golang.py | 157 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 155 insertions(+), 2 deletions(-)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#10 lore raw

Re: [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate types in Go

Quoted 44 lines · Victor Toso

Hi,

Quoted 23 lines · Daniel P. Berrangé
Quoted 10 lines · Victor Toso

This patch handles QAPI alternate types and generates data structures in Go that handles it.

At this moment, there are 5 alternates in qemu/qapi, they are:
* BlockDirtyBitmapMergeSource
* Qcow2OverlapChecks
* BlockdevRef
* BlockdevRefOrNull
* StrOrNull

Quoted 4 lines

Alternate types are similar to Union but without a discriminator that can be used to identify the underlying value on the wire. It is needed to infer it. That can't be easily mapped in Go.

I don't buy that. Given this example:

  type BlockdevRef struct {
        // Options are:
        // * definition (BlockdevOptions): defines a new block device inline
        // * reference (string): references the ID of an existing block device
        Value Any
Quoted 8 lines

}

What is the problem with having this Go struct:

type BlockdevRef struct {
Definition *BlockdevOptions
Reference *string
}

... this is better.

Quoted 7 lines

when deserializing from JSON, we know exactly which one of these two fields to populate. The programmer consuming this can look at which field is non-nil.

When serializing to JSON, we serialize which ever field is non-nil.

Quoted 7 lines
Quoted 3 lines

If both fields are non-nil that's a programmer bug. Either ignore it and only serialize the first non-nil field, or raise an error.

It would be a programmer bug if they set a Value of a type not allowed by Type's spec, but it would be a *runtime* error. Your suggestion is more type safe.

Yep, essentially I'm saying I want the code to enable type checking to be done at compile time instead of runtime, whenever it is possible to achieve that from a technical POV.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#11 lore raw

RFC PATCH 3/8 qapi: golang: Generate qapi's struct types in Go

In reply to Victor Toso

This patch handles QAPI struct types and generates the equivalent types in Go.

At the time of this writing, it generates 375 structures.

The highlights of this implementation are:

1. Generating an Go struct that requires a @base type, the @base type is embedded in this Go structure.

Example: See InetSocketAddress with it's base InetSocketAddressBase

2. Differently from previous two types ('enum' and 'alternate'), the generated QAPI's struct type do not need to implement Marshaler and Unmarshaler interfaces. This generated structures will naturally match with JSON Objects.

3. About the Go struct's fields:

i) They can be either by Value or Reference.

ii) Every field that is marked as optional in the QAPI specification are translated to Reference fields in its Go structure. This design decision is the most straightforward way to check if a given field was set or not.

iii) Mandatory fields are always by Value with the exception of QAPI arrays, which are handled by Reference (to a block of memory) by Go.

iv) All the fields are named with Uppercase due Golang's export convention.

v) In order to avoid any kind of issues when encoding ordecoding, to or from JSON, we mark all fields with its @name and, when it is optional, member, with @omitempty

---
 scripts/qapi/golang.py | 79 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 77 insertions(+), 2 deletions(-)
scripts/qapi/golang.py +77 -2
index 8be31bd902..50e39f8925 100644
@@ -31,7 +31,7 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
 
     def __init__(self, prefix: str):
         super().__init__()
-        self.target = {name: "" for name in ["alternate", "enum", "helper"]}
+        self.target = {name: "" for name in ["alternate", "enum", "helper", "struct"]}
         self.objects_seen = {}
         self.schema = None
         self._docmap = {}
@@ -82,7 +82,31 @@ def visit_object_type(self: QAPISchemaGenGolangVisitor,
                           members: List[QAPISchemaObjectTypeMember],
                           variants: Optional[QAPISchemaVariants]
                           ) -> None:
-        pass
+        # Do not handle anything besides structs
+        if (name == self.schema.the_empty_object_type.name or
+                not isinstance(name, str) or
+                info.defn_meta not in ["struct"]):
+            return
+
+        assert name not in self.objects_seen
+        self.objects_seen[name] = True
+
+        # visit all inner objects as well, they are not going to be
+        # called by python's generator.
+        if variants:
+            for var in variants.variants:
+                assert isinstance(var.type, QAPISchemaObjectType)
+                self.visit_object_type(self,
+                                       var.type.name,
+                                       var.type.info,
+                                       var.type.ifcond,
+                                       var.type.base,
+                                       var.type.local_members,
+                                       var.type.variants)
+
+        doc = self._docmap.get(info.defn_name, None)
+        self.target[info.defn_meta] += qapi_to_golang_struct(name, doc, info,
+                ifcond, features, base, members, variants)
 
     def visit_alternate_type(self: QAPISchemaGenGolangVisitor,
                              name: str,
@@ -276,6 +300,14 @@ def gen_golang(schema: QAPISchema,
     schema.visit(vis)
     vis.write(output_dir)
 
+# Helper function for boxed or self contained structures.
+def generate_struct_type(type_name, args="", doc_struct="") -> str:
+    args =  args if len(args) == 0 else f"\n{args}\n"
+    return f'''
+{doc_struct}
+type {type_name} struct {{{args}}}
+'''
+
 def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str:
     type = qapi_to_go_type_name(members[0].defined_in, "enum")
 
@@ -345,6 +377,46 @@ def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]):
 
     return main, fields
 
+# Helper function that is used for most of QAPI types
+def qapi_to_golang_struct(name: str,
+                          doc: QAPIDoc,
+                          info: Optional[QAPISourceInfo],
+                          ifcond: QAPISchemaIfCond,
+                          features: List[QAPISchemaFeature],
+                          base: Optional[QAPISchemaObjectType],
+                          members: List[QAPISchemaObjectTypeMember],
+                          variants: Optional[QAPISchemaVariants]) -> str:
+
+    type_name = qapi_to_go_type_name(name, info.defn_meta)
+    doc_struct, doc_fields = qapi_to_golang_struct_docs(doc)
+
+    base_fields = ""
+    if base:
+        base_type_name = qapi_to_go_type_name(base.name, base.meta)
+        base_fields = f"\t// Base type for this struct\n\t{base_type_name}\n"
+
+    own_fields = ""
+    for memb in members:
+        field = qapi_to_field_name(memb.name)
+        member_type = qapi_schema_type_to_go_type(memb.type.name)
+
+        # In Golang, we are using "encoding/json" library to Marshal and Unmarshal between
+        # over-the-wire QMP and Golang struct. The field tag 'omitempty' does not behave as
+        # expected for some types with default values and they only way to "ignore by default"
+        # unset fields is to have them as reference in the Struct.
+        # This way, a *bool and *int can be ignored where a bool or int might have been set.
+        isptr = "*" if memb.optional and member_type[0] not in "*[" else ""
+        optional = ",omitempty" if memb.optional else ""
+        fieldtag = '`json:"{name}{optional}"`'.format(name=memb.name,optional=optional)
+
+        field_doc = doc_fields.get(memb.name, "")
+        own_fields += f"\t{field} {isptr}{member_type}{fieldtag}{field_doc}\n"
+
+    all_fields = base_fields if len(base_fields) > 0 else ""
+    all_fields += own_fields[:-1] if len(own_fields) > 0 else ""
+
+    return generate_struct_type(type_name, all_fields, doc_struct)
+
 def qapi_schema_type_to_go_type(type: str) -> str:
     schema_types_to_go = {'str': 'string', 'null': 'nil', 'bool': 'bool',
             'number': 'float64', 'size': 'uint64', 'int': 'int64', 'int8': 'int8',
@@ -364,6 +436,9 @@ def qapi_schema_type_to_go_type(type: str) -> str:
 def qapi_to_field_name_enum(name: str) -> str:
     return name.title().replace("-", "")
 
+def qapi_to_field_name(name: str) -> str:
+    return name.title().replace("_", "").replace("-", "")
+
 def qapi_to_go_type_name(name: str, meta: str) -> str:
     if name.startswith("q_obj_"):
         name = name[6:]
Signature · 1 line
2.35.1
#12 lore raw

RFC PATCH 4/8 qapi: golang: Generate qapi's union types in Go

In reply to Victor Toso

This patch handles QAPI union types and generates the equivalent data structures and methods in Go to handle it.

At the moment of this writing, it generates 67 structures.

The QAPI union type can be summarized by its common members that are defined in a @base struct and a @value. The @value type can vary and depends on @base's field that we call @discriminator. The @discriminator is always a Enum type.

Golang does not have Unions. The generation of QAPI union type in Go with this patch, follows similar approach to what is done for QAPI struct types and QAPI alternate types.

Similarly to Go implementation of QAPI alternate types, we will implement the Marshaler and Unmarshaler interfaces to seamless decode from JSON objects to Golang structs and vice versa.

Similarly to Go implementation of QAPI struct types, we will need to tag @base fields accordingly.

The embedded documentation in Golang's structures and fields are particularly important here, to help developers know what Types to use for @value. Runtime checks too.

---
 scripts/qapi/golang.py | 124 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 119 insertions(+), 5 deletions(-)
scripts/qapi/golang.py +119 -5
index 50e39f8925..0a1bf430ba 100644
@@ -31,7 +31,7 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
 
     def __init__(self, prefix: str):
         super().__init__()
-        self.target = {name: "" for name in ["alternate", "enum", "helper", "struct"]}
+        self.target = {name: "" for name in ["alternate", "enum", "helper", "struct", "union"]}
         self.objects_seen = {}
         self.schema = None
         self._docmap = {}
@@ -82,10 +82,10 @@ def visit_object_type(self: QAPISchemaGenGolangVisitor,
                           members: List[QAPISchemaObjectTypeMember],
                           variants: Optional[QAPISchemaVariants]
                           ) -> None:
-        # Do not handle anything besides structs
+        # Do not handle anything besides struct and unions.
         if (name == self.schema.the_empty_object_type.name or
                 not isinstance(name, str) or
-                info.defn_meta not in ["struct"]):
+                info.defn_meta not in ["struct", "union"]):
             return
 
         assert name not in self.objects_seen
@@ -351,6 +351,93 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str:
 }}
 '''
 
+# Marshal methods for Union types
+def generate_marshal_methods(type: str,
+                             type_dict: Dict[str, str],
+                             discriminator: str = "",
+                             base: str = "") -> str:
+    assert base != ""
+    discriminator = "base." + discriminator
+
+    switch_case_format = '''
+    case {name}:
+        value := {case_type}{{}}
+        if err := json.Unmarshal(data, &value); err != nil {{
+            return err
+        }}
+        s.Value = value'''
+
+    if_supported_types = ""
+    added = {}
+    switch_cases = ""
+    for name in sorted(type_dict):
+        case_type = type_dict[name]
+        isptr = "*" if case_type[0] not in "*[" else ""
+        switch_cases += switch_case_format.format(name = name,
+                                                  case_type = case_type)
+        if case_type not in added:
+            if_supported_types += f'''typestr != "{case_type}" &&\n\t\t'''
+            added[case_type] = True
+
+    marshalfn = f'''
+func (s {type}) MarshalJSON() ([]byte, error) {{
+	base, err := json.Marshal(s.{base})
+	if err != nil {{
+		return nil, err
+	}}
+
+    typestr := fmt.Sprintf("%T", s.Value)
+    typestr = typestr[strings.LastIndex(typestr, ".")+1:]
+
+    // "The branches need not cover all possible enum values"
+    // This means that on Marshal, we can safely ignore empty values
+    if typestr == "<nil>" {{
+        return []byte(base), nil
+    }}
+
+    // Runtime check for supported value types
+    if {if_supported_types[:-6]} {{
+        return nil, errors.New(fmt.Sprintf("Type is not supported: %s", typestr))
+    }}
+	value, err := json.Marshal(s.Value)
+	if err != nil {{
+		return nil, err
+	}}
+
+    // Workaround to avoid checking s.Value being empty
+    if string(value) == "{{}}" {{
+        return []byte(base), nil
+    }}
+
+    // Removes the last '}}' from base and the first '{{' from value, in order to
+    // return a single JSON object.
+    result := fmt.Sprintf("%s,%s", base[:len(base)-1], value[1:])
+    return []byte(result), nil
+}}
+'''
+    unmarshal_base = f'''
+    var base {base}
+    if err := json.Unmarshal(data, &base); err != nil {{
+        return err
+    }}
+    s.{base} = base
+'''
+    unmarshal_default_warn = f'''
+    default:
+        fmt.Println("Failed to decode {type}", {discriminator})'''
+
+    return f'''{marshalfn}
+func (s *{type}) UnmarshalJSON(data []byte) error {{
+    {unmarshal_base}
+    switch {discriminator} {{
+{switch_cases[1:]}
+    {unmarshal_default_warn}
+    }}
+
+    return nil
+}}
+'''
+
 # Takes the documentation object of a specific type and returns
 # that type's documentation followed by its member's docs.
 def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]):
@@ -412,10 +499,37 @@ def qapi_to_golang_struct(name: str,
         field_doc = doc_fields.get(memb.name, "")
         own_fields += f"\t{field} {isptr}{member_type}{fieldtag}{field_doc}\n"
 
+    union_types = {}
+    variant_fields = ""
+    if variants:
+        variant_fields = f"// Value based on @{variants.tag_member.name}, possible types:"
+        for var in variants.variants:
+            if var.type.is_implicit():
+                continue
+
+            name = variants.tag_member._type_name + var.name.title().replace("-", "")
+            union_types[name] = var.type.name
+            variant_fields += f"\n\t// * {var.type.c_unboxed_type()}"
+
+        variant_fields += f"\n\tValue Any"
+
     all_fields = base_fields if len(base_fields) > 0 else ""
     all_fields += own_fields[:-1] if len(own_fields) > 0 else ""
-
-    return generate_struct_type(type_name, all_fields, doc_struct)
+    all_fields += variant_fields if len(variant_fields) > 0 else ""
+
+    unmarshal_fn = ""
+    if info.defn_meta == "union" and variants is not None:
+        # Union's without variants are the Union's base data structure.
+        # e.g: SchemaInfo's base is SchemainfoBase.
+        discriminator = qapi_to_field_name(variants.tag_member.name)
+        base = qapi_to_go_type_name(variants.tag_member.defined_in,
+                                    variants.tag_member.info.defn_meta)
+        unmarshal_fn = generate_marshal_methods(type_name,
+                                                union_types,
+                                                discriminator = discriminator,
+                                                base = base_type_name)
+
+    return generate_struct_type(type_name, all_fields, doc_struct) + unmarshal_fn
 
 def qapi_schema_type_to_go_type(type: str) -> str:
     schema_types_to_go = {'str': 'string', 'null': 'nil', 'bool': 'bool',
Signature · 1 line
2.35.1
#13 lore raw

Re: [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union types in Go

Quoted 5 lines · Victor Toso

This patch handles QAPI union types and generates the equivalent data structures and methods in Go to handle it.

At the moment of this writing, it generates 67 structures.

Quoted 8 lines

The QAPI union type can be summarized by its common members that are defined in a @base struct and a @value. The @value type can vary and depends on @base's field that we call @discriminator. The @discriminator is always a Enum type.

Golang does not have Unions. The generation of QAPI union type in Go with this patch, follows similar approach to what is done for QAPI struct types and QAPI alternate types.

The common way to approach unions in Go is to just use a struct where each union case is an optional field, and declare that only one field must ever be set. ie

  type SocketAddressLegacy struct {
        // Value based on @type, possible types:
        Inet *InetSocketAddressWrapper
        Unix *UnixSocketAddressWrapper
        VSock *VsockSocketAddressWrapper
        FD *StringWrapper
  }

When deserializing from JSON we populate exactly one of the optional fields.

When serializing to JSON process the first field that is non-nil.

Note, you don't actually need to include the discriminator as a field at all, since it is implicitly determined by whichever case is non-nil. Introducing the discriminator as a field just provides the possibility for the programmer to make inconsistent settings, for no gain.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#14 lore raw

Re: [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union types in Go

Hi,

Quoted 19 lines · Daniel P. Berrangé
Quoted 5 lines · Victor Toso

This patch handles QAPI union types and generates the equivalent data structures and methods in Go to handle it.

At the moment of this writing, it generates 67 structures.

Quoted 8 lines

The QAPI union type can be summarized by its common members that are defined in a @base struct and a @value. The @value type can vary and depends on @base's field that we call @discriminator. The @discriminator is always a Enum type.

Golang does not have Unions. The generation of QAPI union type in Go with this patch, follows similar approach to what is done for QAPI struct types and QAPI alternate types.

The common way to approach unions in Go is to just use a struct where each union case is an optional field, and declare that only one field must ever be set. ie

Quoted 7 lines
  type SocketAddressLegacy struct {
        // Value based on @type, possible types:
        Inet *InetSocketAddressWrapper
        Unix *UnixSocketAddressWrapper
        VSock *VsockSocketAddressWrapper
        FD *StringWrapper
  }

Like Alternates, I like this better.

Quoted 11 lines

When deserializing from JSON we populate exactly one of the optional fields.

When serializing to JSON process the first field that is non-nil.

Note, you don't actually need to include the discriminator as a field at all, since it is implicitly determined by whichever case is non-nil. Introducing the discriminator as a field just provides the possibility for the programmer to make inconsistent settings, for no gain.

Sounds reasonable. We still need to implement Marshal/Unmarshal for unknow types (e.g: a new Type for SocketAddressLegacy was introduced in 7.1 and we should be able to know that current qapi-go version can't understand it).

Cheers,
Victor

#15 lore raw

Re: [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union types in Go

Quoted 44 lines · Victor Toso

Hi,

Quoted 19 lines · Daniel P. Berrangé
Quoted 5 lines · Victor Toso

This patch handles QAPI union types and generates the equivalent data structures and methods in Go to handle it.

At the moment of this writing, it generates 67 structures.

Quoted 8 lines

The QAPI union type can be summarized by its common members that are defined in a @base struct and a @value. The @value type can vary and depends on @base's field that we call @discriminator. The @discriminator is always a Enum type.

Golang does not have Unions. The generation of QAPI union type in Go with this patch, follows similar approach to what is done for QAPI struct types and QAPI alternate types.

The common way to approach unions in Go is to just use a struct where each union case is an optional field, and declare that only one field must ever be set. ie

Quoted 7 lines
  type SocketAddressLegacy struct {
        // Value based on @type, possible types:
        Inet *InetSocketAddressWrapper
        Unix *UnixSocketAddressWrapper
        VSock *VsockSocketAddressWrapper
        FD *StringWrapper
  }

Like Alternates, I like this better.

Quoted 11 lines

When deserializing from JSON we populate exactly one of the optional fields.

When serializing to JSON process the first field that is non-nil.

Note, you don't actually need to include the discriminator as a field at all, since it is implicitly determined by whichever case is non-nil. Introducing the discriminator as a field just provides the possibility for the programmer to make inconsistent settings, for no gain.

Quoted 4 lines

Sounds reasonable. We still need to implement Marshal/Unmarshal for unknow types (e.g: a new Type for SocketAddressLegacy was introduced in 7.1 and we should be able to know that current qapi-go version can't understand it).

If there's a new type seen on the wire then, the easy option is to just not deserialize it at all. Return a SocketAddressLegacy struct with all fields nil, or perhaps return a full 'error', since this is likely to be significant functionalproblem for the application.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#16 lore raw

RFC PATCH 5/8 qapi: golang: Generate qapi's event types in Go

In reply to Victor Toso

This patch handles QAPI event types and generates data structures in Go that handles it.

At the moment of this writing, it generates 51 structures (49 events)

In Golang, each event is handled as a Go structure and there is no big difference, in the Go generated code, between what is a QAPI event type and what is a QAPI struct.

Each QAPI event has the suffix 'Event' in its Golang data structure and contains the fields, mandatory and optional, that can be sent or received.

In addition, there are two structures added to handle QAPI specification for event types: 'Event' and 'EventBase'.

'EventBase' contains @Name and @Timestamp fields and then 'Event' extends 'EventBase' with an @Arg field of type 'Any'.

The 'Event' type implements the Unmarshaler to decode the QMP JSON Object into the correct Golang (event) struct. The goal here is to facilitate receiving Events.

A TODO for this type is to implement Marshaler for 'Event'. It'll containt runtime checks to validate before transforming the struct into a JSON Object.

Example:

```go
    qmpMsg := `{
    "event" : "MIGRATION",
    "timestamp":{
        "seconds": 1432121972,
        "microseconds": 744001
    },
    "data":{
        "status": "completed"
    }
}`

    e := Event{}
    _ = json.Unmarshal([]byte(qmpMsg), &e)
    // e.Name == "MIGRATION"
    // e.Arg.(MigrationEvent).Status == MigrationStatusCompleted
```
---
 scripts/qapi/golang.py | 92 ++++++++++++++++++++++++++++++++++++++----
 1 file changed, 84 insertions(+), 8 deletions(-)
scripts/qapi/golang.py +84 -8
index 0a1bf430ba..3bb66d07c7 100644
@@ -31,9 +31,10 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
 
     def __init__(self, prefix: str):
         super().__init__()
-        self.target = {name: "" for name in ["alternate", "enum", "helper", "struct", "union"]}
+        self.target = {name: "" for name in ["alternate", "enum", "event", "helper", "struct", "union"]}
         self.objects_seen = {}
         self.schema = None
+        self.events = {}
         self._docmap = {}
         self.golang_package_name = "qapi"
 
@@ -57,6 +58,24 @@ def visit_begin(self, schema):
     def visit_end(self):
         self.schema = None
 
+        # EventBase and Event are not specified in the QAPI schema,
+        # so we need to generate it ourselves.
+        self.target["event"] += '''
+type EventBase struct {
+    Name      string `json:"event"`
+    Timestamp struct {
+        Seconds      int64 `json:"seconds"`
+        Microseconds int64 `json:"microseconds"`
+    } `json:"timestamp"`
+}
+
+type Event struct {
+    EventBase
+    Arg       Any    `json:"data,omitempty"`
+}
+'''
+        self.target["event"] += generate_marshal_methods('Event', self.events)
+
         self.target["helper"] += '''
 // Creates a decoder that errors on unknown Fields
 // Returns true if successfully decoded @from string @into type
@@ -279,7 +298,28 @@ def visit_command(self,
         pass
 
     def visit_event(self, name, info, ifcond, features, arg_type, boxed):
-        pass
+        assert name == info.defn_name
+        type_name = qapi_to_go_type_name(name, info.defn_meta)
+        self.events[name] = type_name
+
+        doc = self._docmap.get(name, None)
+        self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False
+        content = ""
+        if self_contained:
+            doc_struct, _ = qapi_to_golang_struct_docs(doc)
+            content = generate_struct_type(type_name, "", doc_struct)
+        else:
+            assert isinstance(arg_type, QAPISchemaObjectType)
+            content = qapi_to_golang_struct(name,
+                                            doc,
+                                            arg_type.info,
+                                            arg_type.ifcond,
+                                            arg_type.features,
+                                            arg_type.base,
+                                            arg_type.members,
+                                            arg_type.variants)
+
+        self.target["event"] += content
 
     def write(self, output_dir: str) -> None:
         for module_name, content in self.target.items():
@@ -351,15 +391,41 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str:
 }}
 '''
 
-# Marshal methods for Union types
+# Marshal methods for Event and Union types
 def generate_marshal_methods(type: str,
                              type_dict: Dict[str, str],
                              discriminator: str = "",
                              base: str = "") -> str:
-    assert base != ""
-    discriminator = "base." + discriminator
-
-    switch_case_format = '''
+    type_is_union = False
+    json_field = ""
+    struct_field = ""
+    if type == "Event":
+        base = type + "Base"
+        discriminator = "base.Name"
+        struct_field = "Arg"
+        json_field = "data"
+    else:
+        assert base != ""
+        discriminator = "base." + discriminator
+        type_is_union = True
+
+    switch_case_format = ""
+    if not type_is_union:
+        switch_case_format = '''
+    case "{name}":
+        tmp := struct {{
+            Value {isptr}{case_type} `json:"{json_field},omitempty"`
+        }}{{}}
+        if err := json.Unmarshal(data, &tmp); err != nil {{
+            return err
+        }}
+        if tmp.Value == nil {{
+            s.{struct_field} = nil
+        }} else {{
+            s.{struct_field} = {isptr}tmp.Value
+        }}'''
+    else:
+        switch_case_format = '''
     case {name}:
         value := {case_type}{{}}
         if err := json.Unmarshal(data, &value); err != nil {{
@@ -374,12 +440,17 @@ def generate_marshal_methods(type: str,
         case_type = type_dict[name]
         isptr = "*" if case_type[0] not in "*[" else ""
         switch_cases += switch_case_format.format(name = name,
+                                                  struct_field = struct_field,
+                                                  json_field = json_field,
+                                                  isptr = isptr,
                                                   case_type = case_type)
         if case_type not in added:
             if_supported_types += f'''typestr != "{case_type}" &&\n\t\t'''
             added[case_type] = True
 
-    marshalfn = f'''
+    marshalfn = ""
+    if type_is_union:
+        marshalfn = f'''
 func (s {type}) MarshalJSON() ([]byte, error) {{
 	base, err := json.Marshal(s.{base})
 	if err != nil {{
@@ -564,4 +635,9 @@ def qapi_to_go_type_name(name: str, meta: str) -> str:
     words = [word for word in name.replace("_", "-").split("-")]
     name = words[0].title() if words[0].islower() or words[0].isupper() else words[0]
     name += ''.join(word.title() for word in words[1:])
+
+    if meta == "event":
+        name = name[:-3] if name.endswith("Arg") else name
+        name += meta.title()
+
     return name
Signature · 1 line
2.35.1
#17 lore raw

Re: [RFC PATCH v1 5/8] qapi: golang: Generate qapi's event types in Go

Quoted 13 lines · Victor Toso

This patch handles QAPI event types and generates data structures in Go that handles it.

At the moment of this writing, it generates 51 structures (49 events)

In Golang, each event is handled as a Go structure and there is no big difference, in the Go generated code, between what is a QAPI event type and what is a QAPI struct.

Each QAPI event has the suffix 'Event' in its Golang data structure and contains the fields, mandatory and optional, that can be sent or received.

Quoted 5 lines

In addition, there are two structures added to handle QAPI specification for event types: 'Event' and 'EventBase'.

'EventBase' contains @Name and @Timestamp fields and then 'Event' extends 'EventBase' with an @Arg field of type 'Any'.

Again, I don't think we should need to use an Any type here.

Rather than

  type EventBase struct {
        Name      string `json:"event"`
        Timestamp struct {
                Seconds      int64 `json:"seconds"`
                Microseconds int64 `json:"microseconds"`
        } `json:"timestamp"`
  }

  type Event struct {
        EventBase
        Arg Any `json:"data,omitempty"`
  }

  type ShutdownEvent struct {
        Guest  bool          `json:"guest"`
        Reason ShutdownCause `json:"reason"`
  }

I think we should just embed EventBase directly in each specific event eg

  type Event struct {
        Name      string `json:"event"`
        Timestamp struct {
                Seconds      int64 `json:"seconds"`
                Microseconds int64 `json:"microseconds"`
        } `json:"timestamp"`
  }

  type ShutdownEvent struct {
        Event Event
        Guest  bool          `json:"guest"`
        Reason ShutdownCause `json:"reason"`
  }

Or perhaps better still, use an interface

  type Event interface {
      GetName() string
      GetTimestamp() string
  }

  type Timestamp struct {
      Seconds      int64 `json:"seconds"`
      Microseconds int64 `json:"microseconds"`
  }

  type ShutdownEvent struct {
        Timestamp Timestamp  `json:"timestamp"`
        Guest  bool          `json:"guest"`
        Reason ShutdownCause `json:"reason"`
  }

  func (ev *ShutdownEvent) GetName() string {
        return "SHUTDOWN"
  }

That way you can define public APIs taking 'Event' as a type, and impls of the events can be passed directly in/out.

Similar comment for the Command type.

Quoted 184 lines

The 'Event' type implements the Unmarshaler to decode the QMP JSON Object into the correct Golang (event) struct. The goal here is to facilitate receiving Events.

A TODO for this type is to implement Marshaler for 'Event'. It'll containt runtime checks to validate before transforming the struct into a JSON Object.

Example:

```go
    qmpMsg := `{
    "event" : "MIGRATION",
    "timestamp":{
        "seconds": 1432121972,
        "microseconds": 744001
    },
    "data":{
        "status": "completed"
    }
}`

    e := Event{}
    _ = json.Unmarshal([]byte(qmpMsg), &e)
    // e.Name == "MIGRATION"
    // e.Arg.(MigrationEvent).Status == MigrationStatusCompleted
```
---
 scripts/qapi/golang.py | 92 ++++++++++++++++++++++++++++++++++++++----
 1 file changed, 84 insertions(+), 8 deletions(-)
scripts/qapi/golang.py +84 -8
index 0a1bf430ba..3bb66d07c7 100644
@@ -31,9 +31,10 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
 
     def __init__(self, prefix: str):
         super().__init__()
-        self.target = {name: "" for name in ["alternate", "enum", "helper", "struct", "union"]}
+        self.target = {name: "" for name in ["alternate", "enum", "event", "helper", "struct", "union"]}
         self.objects_seen = {}
         self.schema = None
+        self.events = {}
         self._docmap = {}
         self.golang_package_name = "qapi"
 
@@ -57,6 +58,24 @@ def visit_begin(self, schema):
     def visit_end(self):
         self.schema = None
 
+        # EventBase and Event are not specified in the QAPI schema,
+        # so we need to generate it ourselves.
+        self.target["event"] += '''
+type EventBase struct {
+    Name      string `json:"event"`
+    Timestamp struct {
+        Seconds      int64 `json:"seconds"`
+        Microseconds int64 `json:"microseconds"`
+    } `json:"timestamp"`
+}
+
+type Event struct {
+    EventBase
+    Arg       Any    `json:"data,omitempty"`
+}
+'''
+        self.target["event"] += generate_marshal_methods('Event', self.events)
+
         self.target["helper"] += '''
 // Creates a decoder that errors on unknown Fields
 // Returns true if successfully decoded @from string @into type
@@ -279,7 +298,28 @@ def visit_command(self,
         pass
 
     def visit_event(self, name, info, ifcond, features, arg_type, boxed):
-        pass
+        assert name == info.defn_name
+        type_name = qapi_to_go_type_name(name, info.defn_meta)
+        self.events[name] = type_name
+
+        doc = self._docmap.get(name, None)
+        self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False
+        content = ""
+        if self_contained:
+            doc_struct, _ = qapi_to_golang_struct_docs(doc)
+            content = generate_struct_type(type_name, "", doc_struct)
+        else:
+            assert isinstance(arg_type, QAPISchemaObjectType)
+            content = qapi_to_golang_struct(name,
+                                            doc,
+                                            arg_type.info,
+                                            arg_type.ifcond,
+                                            arg_type.features,
+                                            arg_type.base,
+                                            arg_type.members,
+                                            arg_type.variants)
+
+        self.target["event"] += content
 
     def write(self, output_dir: str) -> None:
         for module_name, content in self.target.items():
@@ -351,15 +391,41 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str:
 }}
 '''
 
-# Marshal methods for Union types
+# Marshal methods for Event and Union types
 def generate_marshal_methods(type: str,
                              type_dict: Dict[str, str],
                              discriminator: str = "",
                              base: str = "") -> str:
-    assert base != ""
-    discriminator = "base." + discriminator
-
-    switch_case_format = '''
+    type_is_union = False
+    json_field = ""
+    struct_field = ""
+    if type == "Event":
+        base = type + "Base"
+        discriminator = "base.Name"
+        struct_field = "Arg"
+        json_field = "data"
+    else:
+        assert base != ""
+        discriminator = "base." + discriminator
+        type_is_union = True
+
+    switch_case_format = ""
+    if not type_is_union:
+        switch_case_format = '''
+    case "{name}":
+        tmp := struct {{
+            Value {isptr}{case_type} `json:"{json_field},omitempty"`
+        }}{{}}
+        if err := json.Unmarshal(data, &tmp); err != nil {{
+            return err
+        }}
+        if tmp.Value == nil {{
+            s.{struct_field} = nil
+        }} else {{
+            s.{struct_field} = {isptr}tmp.Value
+        }}'''
+    else:
+        switch_case_format = '''
     case {name}:
         value := {case_type}{{}}
         if err := json.Unmarshal(data, &value); err != nil {{
@@ -374,12 +440,17 @@ def generate_marshal_methods(type: str,
         case_type = type_dict[name]
         isptr = "*" if case_type[0] not in "*[" else ""
         switch_cases += switch_case_format.format(name = name,
+                                                  struct_field = struct_field,
+                                                  json_field = json_field,
+                                                  isptr = isptr,
                                                   case_type = case_type)
         if case_type not in added:
             if_supported_types += f'''typestr != "{case_type}" &&\n\t\t'''
             added[case_type] = True
 
-    marshalfn = f'''
+    marshalfn = ""
+    if type_is_union:
+        marshalfn = f'''
 func (s {type}) MarshalJSON() ([]byte, error) {{
 	base, err := json.Marshal(s.{base})
 	if err != nil {{
@@ -564,4 +635,9 @@ def qapi_to_go_type_name(name: str, meta: str) -> str:
     words = [word for word in name.replace("_", "-").split("-")]
     name = words[0].title() if words[0].islower() or words[0].isupper() else words[0]
     name += ''.join(word.title() for word in words[1:])
+
+    if meta == "event":
+        name = name[:-3] if name.endswith("Arg") else name
+        name += meta.title()
+
     return name
Signature · 1 line
2.35.1

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#18 lore raw

Re: [RFC PATCH v1 5/8] qapi: golang: Generate qapi's event types in Go

Hi,

Quoted 84 lines · Daniel P. Berrangé
Quoted 13 lines · Victor Toso

This patch handles QAPI event types and generates data structures in Go that handles it.

At the moment of this writing, it generates 51 structures (49 events)

In Golang, each event is handled as a Go structure and there is no big difference, in the Go generated code, between what is a QAPI event type and what is a QAPI struct.

Each QAPI event has the suffix 'Event' in its Golang data structure and contains the fields, mandatory and optional, that can be sent or received.

Quoted 5 lines

In addition, there are two structures added to handle QAPI specification for event types: 'Event' and 'EventBase'.

'EventBase' contains @Name and @Timestamp fields and then 'Event' extends 'EventBase' with an @Arg field of type 'Any'.

Again, I don't think we should need to use an Any type here.

Rather than

  type EventBase struct {
        Name      string `json:"event"`
        Timestamp struct {
                Seconds      int64 `json:"seconds"`
                Microseconds int64 `json:"microseconds"`
        } `json:"timestamp"`
  }

  type Event struct {
        EventBase
        Arg Any `json:"data,omitempty"`
  }

  type ShutdownEvent struct {
        Guest  bool          `json:"guest"`
        Reason ShutdownCause `json:"reason"`
  }

I think we should just embed EventBase directly in each specific event eg

  type Event struct {
        Name      string `json:"event"`
        Timestamp struct {
                Seconds      int64 `json:"seconds"`
                Microseconds int64 `json:"microseconds"`
        } `json:"timestamp"`
  }

  type ShutdownEvent struct {
        Event Event
        Guest  bool          `json:"guest"`
        Reason ShutdownCause `json:"reason"`
  }

Or perhaps better still, use an interface

  type Event interface {
      GetName() string
      GetTimestamp() string
  }

  type Timestamp struct {
      Seconds      int64 `json:"seconds"`
      Microseconds int64 `json:"microseconds"`
  }

  type ShutdownEvent struct {
        Timestamp Timestamp  `json:"timestamp"`
        Guest  bool          `json:"guest"`
        Reason ShutdownCause `json:"reason"`
  }

  func (ev *ShutdownEvent) GetName() string {
        return "SHUTDOWN"
  }
Quoted 4 lines

That way you can define public APIs taking 'Event' as a type, and impls of the events can be passed directly in/out.

Similar comment for the Command type.

Yeah, if we are removing Any for Unions and Alternates, we can sure do the same for Events and Commands.

Quoted 178 lines
Quoted 178 lines

The 'Event' type implements the Unmarshaler to decode the QMP JSON Object into the correct Golang (event) struct. The goal here is to facilitate receiving Events.

A TODO for this type is to implement Marshaler for 'Event'. It'll containt runtime checks to validate before transforming the struct into a JSON Object.

Example:

```go
    qmpMsg := `{
    "event" : "MIGRATION",
    "timestamp":{
        "seconds": 1432121972,
        "microseconds": 744001
    },
    "data":{
        "status": "completed"
    }
}`

    e := Event{}
    _ = json.Unmarshal([]byte(qmpMsg), &e)
    // e.Name == "MIGRATION"
    // e.Arg.(MigrationEvent).Status == MigrationStatusCompleted
```
---
 scripts/qapi/golang.py | 92 ++++++++++++++++++++++++++++++++++++++----
 1 file changed, 84 insertions(+), 8 deletions(-)
scripts/qapi/golang.py +84 -8
index 0a1bf430ba..3bb66d07c7 100644
@@ -31,9 +31,10 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
 
     def __init__(self, prefix: str):
         super().__init__()
-        self.target = {name: "" for name in ["alternate", "enum", "helper", "struct", "union"]}
+        self.target = {name: "" for name in ["alternate", "enum", "event", "helper", "struct", "union"]}
         self.objects_seen = {}
         self.schema = None
+        self.events = {}
         self._docmap = {}
         self.golang_package_name = "qapi"
 
@@ -57,6 +58,24 @@ def visit_begin(self, schema):
     def visit_end(self):
         self.schema = None
 
+        # EventBase and Event are not specified in the QAPI schema,
+        # so we need to generate it ourselves.
+        self.target["event"] += '''
+type EventBase struct {
+    Name      string `json:"event"`
+    Timestamp struct {
+        Seconds      int64 `json:"seconds"`
+        Microseconds int64 `json:"microseconds"`
+    } `json:"timestamp"`
+}
+
+type Event struct {
+    EventBase
+    Arg       Any    `json:"data,omitempty"`
+}
+'''
+        self.target["event"] += generate_marshal_methods('Event', self.events)
+
         self.target["helper"] += '''
 // Creates a decoder that errors on unknown Fields
 // Returns true if successfully decoded @from string @into type
@@ -279,7 +298,28 @@ def visit_command(self,
         pass
 
     def visit_event(self, name, info, ifcond, features, arg_type, boxed):
-        pass
+        assert name == info.defn_name
+        type_name = qapi_to_go_type_name(name, info.defn_meta)
+        self.events[name] = type_name
+
+        doc = self._docmap.get(name, None)
+        self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False
+        content = ""
+        if self_contained:
+            doc_struct, _ = qapi_to_golang_struct_docs(doc)
+            content = generate_struct_type(type_name, "", doc_struct)
+        else:
+            assert isinstance(arg_type, QAPISchemaObjectType)
+            content = qapi_to_golang_struct(name,
+                                            doc,
+                                            arg_type.info,
+                                            arg_type.ifcond,
+                                            arg_type.features,
+                                            arg_type.base,
+                                            arg_type.members,
+                                            arg_type.variants)
+
+        self.target["event"] += content
 
     def write(self, output_dir: str) -> None:
         for module_name, content in self.target.items():
@@ -351,15 +391,41 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str:
 }}
 '''
 
-# Marshal methods for Union types
+# Marshal methods for Event and Union types
 def generate_marshal_methods(type: str,
                              type_dict: Dict[str, str],
                              discriminator: str = "",
                              base: str = "") -> str:
-    assert base != ""
-    discriminator = "base." + discriminator
-
-    switch_case_format = '''
+    type_is_union = False
+    json_field = ""
+    struct_field = ""
+    if type == "Event":
+        base = type + "Base"
+        discriminator = "base.Name"
+        struct_field = "Arg"
+        json_field = "data"
+    else:
+        assert base != ""
+        discriminator = "base." + discriminator
+        type_is_union = True
+
+    switch_case_format = ""
+    if not type_is_union:
+        switch_case_format = '''
+    case "{name}":
+        tmp := struct {{
+            Value {isptr}{case_type} `json:"{json_field},omitempty"`
+        }}{{}}
+        if err := json.Unmarshal(data, &tmp); err != nil {{
+            return err
+        }}
+        if tmp.Value == nil {{
+            s.{struct_field} = nil
+        }} else {{
+            s.{struct_field} = {isptr}tmp.Value
+        }}'''
+    else:
+        switch_case_format = '''
     case {name}:
         value := {case_type}{{}}
         if err := json.Unmarshal(data, &value); err != nil {{
@@ -374,12 +440,17 @@ def generate_marshal_methods(type: str,
         case_type = type_dict[name]
         isptr = "*" if case_type[0] not in "*[" else ""
         switch_cases += switch_case_format.format(name = name,
+                                                  struct_field = struct_field,
+                                                  json_field = json_field,
+                                                  isptr = isptr,
                                                   case_type = case_type)
         if case_type not in added:
             if_supported_types += f'''typestr != "{case_type}" &&\n\t\t'''
             added[case_type] = True
 
-    marshalfn = f'''
+    marshalfn = ""
+    if type_is_union:
+        marshalfn = f'''
 func (s {type}) MarshalJSON() ([]byte, error) {{
 	base, err := json.Marshal(s.{base})
 	if err != nil {{
@@ -564,4 +635,9 @@ def qapi_to_go_type_name(name: str, meta: str) -> str:
     words = [word for word in name.replace("_", "-").split("-")]
     name = words[0].title() if words[0].islower() or words[0].isupper() else words[0]
     name += ''.join(word.title() for word in words[1:])
+
+    if meta == "event":
+        name = name[:-3] if name.endswith("Arg") else name
+        name += meta.title()
+
Quoted 12 lines
Quoted 5 lines
scripts/qapi/golang.py +0 -0
     return name
Signature · 1 line
2.35.1

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Cheers,
Victor

#19 lore raw

RFC PATCH 6/8 qapi: golang: Generate qapi's command types in Go

In reply to Victor Toso

This patch handles QAPI command types and generates data structures in Go that decodes from QMP JSON Object to Go data structure and vice versa.

At the time of this writing, it generates 210 structures (208 commands)

This is very similar to previous commit, that handles QAPI union types in Go.

Each QAPI command will generate a Go struct with the suffix 'Command'. Its fields, if any, are the mandatory or optional arguments defined in the QAPI command.

Simlar to Event, this patch adds two structures to handle QAPI specification for command types: 'Command' and 'CommandBase'.

'CommandBase' contains @Id and @Name. 'Command' extends 'CommandBase' with an @Arg field of type 'Any'.

The 'Command' type implements the Unmarshaler to decode QMP JSON Object into the correct Golang (command) struct.

Marshal example:

```go
	cmdArg := SetPasswordCommand{}
	cmdArg.Protocol = DisplayProtocolVnc
	cmdArg.Password = "secret"
	cmd := Command{}
	cmd.Name = "set_password"
	cmd.Arg = cmdArg

	b, _ := json.Marshal(&cmd)
	// string(b) == `{"execute":"set_password","arguments":{"protocol":"vnc","password":"secret"}}`
```

Unmarshal example:

```go
	qmpCommand := `
{
	"execute": "set_password",
	"arguments":{
		"protocol": "vnc",
		"password": "secret"
	}
}`
	cmd := Command{}
	_ = json.Unmarshal([]byte(qmpCommand), &cmd)
	// cmd.Name == "set_password"
	// cmd1.Arg.(SetPasswordCommand).Protocol == DisplayProtocolVnc
```
---
 scripts/qapi/golang.py | 49 ++++++++++++++++++++++++++++++++++++++----
 1 file changed, 45 insertions(+), 4 deletions(-)
scripts/qapi/golang.py +45 -4
index 3bb66d07c7..0b9c19babb 100644
@@ -31,10 +31,11 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor):
 
     def __init__(self, prefix: str):
         super().__init__()
-        self.target = {name: "" for name in ["alternate", "enum", "event", "helper", "struct", "union"]}
+        self.target = {name: "" for name in ["alternate", "command", "enum", "event", "helper", "struct", "union"]}
         self.objects_seen = {}
         self.schema = None
         self.events = {}
+        self.commands = {}
         self._docmap = {}
         self.golang_package_name = "qapi"
 
@@ -76,6 +77,19 @@ def visit_end(self):
 '''
         self.target["event"] += generate_marshal_methods('Event', self.events)
 
+        self.target["command"] += '''
+type CommandBase struct {
+    Id   string `json:"id,omitempty"`
+    Name string `json:"execute"`
+}
+
+type Command struct {
+    CommandBase
+    Arg Any    `json:"arguments,omitempty"`
+}
+'''
+        self.target["command"] += generate_marshal_methods('Command', self.commands)
+
         self.target["helper"] += '''
 // Creates a decoder that errors on unknown Fields
 // Returns true if successfully decoded @from string @into type
@@ -295,7 +309,29 @@ def visit_command(self,
                       allow_oob: bool,
                       allow_preconfig: bool,
                       coroutine: bool) -> None:
-        pass
+        assert name == info.defn_name
+        type_name = qapi_to_go_type_name(name, info.defn_meta)
+        self.commands[name] = type_name
+
+        doc = self._docmap.get(name, None)
+        self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False
+        content = ""
+        if boxed or self_contained:
+            args = "" if not arg_type else "\n" + arg_type.name
+            doc_struct, _ = qapi_to_golang_struct_docs(doc)
+            content = generate_struct_type(type_name, args, doc_struct)
+        else:
+            assert isinstance(arg_type, QAPISchemaObjectType)
+            content = qapi_to_golang_struct(name,
+                                            doc,
+                                            arg_type.info,
+                                            arg_type.ifcond,
+                                            arg_type.features,
+                                            arg_type.base,
+                                            arg_type.members,
+                                            arg_type.variants)
+
+        self.target["command"] += content
 
     def visit_event(self, name, info, ifcond, features, arg_type, boxed):
         assert name == info.defn_name
@@ -391,7 +427,7 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str:
 }}
 '''
 
-# Marshal methods for Event and Union types
+# Marshal methods for Event, Commad and Union types
 def generate_marshal_methods(type: str,
                              type_dict: Dict[str, str],
                              discriminator: str = "",
@@ -404,6 +440,11 @@ def generate_marshal_methods(type: str,
         discriminator = "base.Name"
         struct_field = "Arg"
         json_field = "data"
+    elif type == "Command":
+        base = type + "Base"
+        discriminator = "base.Name"
+        struct_field = "Arg"
+        json_field = "arguments"
     else:
         assert base != ""
         discriminator = "base." + discriminator
@@ -636,7 +677,7 @@ def qapi_to_go_type_name(name: str, meta: str) -> str:
     name = words[0].title() if words[0].islower() or words[0].isupper() else words[0]
     name += ''.join(word.title() for word in words[1:])
 
-    if meta == "event":
+    if meta == "event" or meta == "command":
         name = name[:-3] if name.endswith("Arg") else name
         name += meta.title()
 
Signature · 1 line
2.35.1
#20 lore raw

RFC PATCH 7/8 qapi: golang: Add CommandResult type to Go

In reply to Victor Toso

This patch adds a struct type in Go that will handle return values for QAPI's command types.

The return value of a Command is, encouraged to be, QAPI's complext types or an Array of those.

The 'CommandResult' type acts in similar fashion to 'Event' and 'Command', in order to map the right return data structure based on the issued 'Command'.

This patch also adds a syntax sugar method to 'Command' to return the 'CommandResult' struct to use when receiving the return data.

Example:

```go
    cmd := Command{}
    cmd.Name = `query-tpm-models`
    // bytes, _ := json.Marshal(&cmd)
    // send bytes ...
    received := `{"return":["tpm-tis","tpm-crb","tpm-spapr"]}`
    cmdRet := cmd.GetReturnType()
    _ = json.Unmarshal([]byte(received), &cmdRet)
    // cmdRet.Value.([]TpmModel)[2] == TpmModelTpmSpapr
```
---
 scripts/qapi/golang.py | 42 ++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 40 insertions(+), 2 deletions(-)
scripts/qapi/golang.py +40 -2
index 0b9c19babb..5d3395514d 100644
@@ -36,6 +36,7 @@ def __init__(self, prefix: str):
         self.schema = None
         self.events = {}
         self.commands = {}
+        self.command_results = {}
         self._docmap = {}
         self.golang_package_name = "qapi"
 
@@ -90,6 +91,32 @@ def visit_end(self):
 '''
         self.target["command"] += generate_marshal_methods('Command', self.commands)
 
+        self.target["command"] += '''
+type CommandResult struct {
+	CommandBase
+	Value       Any `json:"return,omitempty"`
+}
+
+func (s Command) GetReturnType() CommandResult {
+	return CommandResult{
+		CommandBase: s.CommandBase,
+	}
+}
+
+// In order to evaluate nil value to empty JSON object
+func (s *CommandResult) MarshalJSON() ([]byte, error) {
+	if s.Value == nil {
+		return []byte(`{"return":{}}`), nil
+	}
+	tmp := struct {
+		Value Any `json:"return"`
+	}{Value: s.Value}
+
+	return json.Marshal(&tmp)
+}
+'''
+        self.target["command"] += generate_marshal_methods('CommandResult', self.command_results)
+
         self.target["helper"] += '''
 // Creates a decoder that errors on unknown Fields
 // Returns true if successfully decoded @from string @into type
@@ -312,6 +339,9 @@ def visit_command(self,
         assert name == info.defn_name
         type_name = qapi_to_go_type_name(name, info.defn_meta)
         self.commands[name] = type_name
+        if ret_type:
+            ret_type_name = qapi_schema_type_to_go_type(ret_type.name)
+            self.command_results[name] = ret_type_name
 
         doc = self._docmap.get(name, None)
         self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False
@@ -445,6 +475,11 @@ def generate_marshal_methods(type: str,
         discriminator = "base.Name"
         struct_field = "Arg"
         json_field = "arguments"
+    elif type == "CommandResult":
+        base = "CommandBase"
+        discriminator = "s.Name"
+        struct_field = "Value"
+        json_field = "return"
     else:
         assert base != ""
         discriminator = "base." + discriminator
@@ -527,14 +562,17 @@ def generate_marshal_methods(type: str,
     return []byte(result), nil
 }}
 '''
-    unmarshal_base = f'''
+    unmarshal_base = ""
+    unmarshal_default_warn = ""
+    if type != "CommandResult":
+        unmarshal_base = f'''
     var base {base}
     if err := json.Unmarshal(data, &base); err != nil {{
         return err
     }}
     s.{base} = base
 '''
-    unmarshal_default_warn = f'''
+        unmarshal_default_warn = f'''
     default:
         fmt.Println("Failed to decode {type}", {discriminator})'''
 
Signature · 1 line
2.35.1
#21 lore raw

RFC PATCH 8/8 qapi: golang: document skip function visit_array_types

In reply to Victor Toso
---
 scripts/qapi/golang.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
scripts/qapi/golang.py +6 -1
index 5d3395514d..9a775d0691 100644
@@ -321,7 +321,12 @@ def visit_enum_type(self: QAPISchemaGenGolangVisitor,
 '''
 
     def visit_array_type(self, name, info, ifcond, element_type):
-        pass
+        # TLDR: We don't need to any extra boilerplate in Go to handle Arrays.
+        #
+        # This function is implemented just to be sure that:
+        # 1. Every array type ends with List
+        # 2. Every array type's element is the array type without 'List'
+        assert name.endswith("List") and name[:-4] == element_type.name
 
     def visit_command(self,
                       name: str,
Signature · 1 line
2.35.1
#22 lore raw
In reply to Victor Toso
Quoted 2 lines · Victor Toso

Thanks for taking a look, let me know if you have questions, ideas or suggestions.

Full disclosure: I have only given the actual implementation a very cursory look so far, and I've focused on the generated Go API instead.

Overall things look pretty good.

One concern that I have is about naming struct members: things like SpiceInfo.MouseMode and most others are translated from the QAPI schema exactly the way you'd expect them, but for example ChardevCommon.Logappend doesn't look quite right. Of course there's no way to programmatically figure out what to capitalize, but maybe there's room for adding this kind of information in the form of additional annotations or something like that? Same for the various structs or members that have unexpectedly-capitalized "Tls" or "Vnc" in them.

To be clear, I don't think the above is a blocker - just something to be aware of, and think about.

My biggest concern is about the interface offered for commands.

Based on the example you have in the README and how commands are defined, invoking (a simplified version of) the trace-event-get-state command would look like

  cmd := Command{
      Name: "trace-event-get-state",
      Arg: TraceEventGetStateCommand{
          Name: "qemu_memalign",
      },
  }
  qmp_input, _ := json.Marshal(&cmd)
  // qmp_input now contains
  //   {"execute":"trace-event-get-state","arguments":{"name":"qemu_memalign"}}
  // do something with it

qmp_output :=
([]byte)({"return":{"name":"qemu_memalign","state":"disabled"}})
ret := cmd.GetReturnType()
_ = json.Unmarshal(qmp_output, &ret)
// ret is a CommandResult instance whose Value member can be cast // to a TraceEventInfo struct

First of all, from an application's point of view there are way too many steps involved: performing this operation should really be as simple as

ret, _ := qmp.TraceEventGetState("qemu_memalign") // ret is a TraceEventInfo instance

That's the end state we should be working towards.

Of course that assumes that the "qmp" object knows where the QMP socket is, knows how to talk the QMP protocol, transparently deals with serializing and deserializing data... Plus, in some case you might want to deal with the wire transfer yourself in an application-specific manner. So it makes sense to have the basic building blocks available and then build the more ergonomic SDK on top of that - with only the first part being in scope for this series.

Even with that in mind, the current interface is IMO problematic because of its almost complete lack of type safety. Both Command.Arg and CommandResult.Value are of type Any and CommandBase.Name, which is used to drive the JSON unmarshal logic as well as ending up on the wire when executing a command, is just a plain string.

I think the low-level interface should look more like

  cmd := TraceEventGetStateCommand{
      Name: "qemu_memalign",
  }
  qmp_input, _ := json.Marshal(&cmd)
  // qmp_input looks the same as before

qmp_output :=
([]byte)({"return":{"name":"qemu_memalign","state":"disabled"}})
ret := TraceEventInfo{}
_ = json.Unmarshal(qmp_output, &ret)
// ret is a TraceEventInfo instance

The advantages over the current implementation is that the compiler will prevent you from doing something silly like passing the wrong set of arguments to a commmand, and that the application has to explicitly spell out what kind of object it expects to get as output.

I'm attaching an incomplete implementation that I used for playing around. It's obviously too simplistic, but hopefully it will help illustrate my point.

Dealing with errors and commands that don't have a return value might require us to have generic CommandResult wrapper after all, but we should really try as hard as we can to stick to type safe interfaces.

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization

package main

import (

	"encoding/json"
	"fmt"

)

type TraceEventGetStateCommand struct {
Name string json:"name"
}

func (self *TraceEventGetStateCommand) MarshalJSON() ([]byte, error) {
	type Arguments TraceEventGetStateCommand
	return json.Marshal(&struct {
		Execute   string     `json:"execute"`
		Arguments *Arguments `json:"arguments"`
	}{
		Execute:   "trace-event-get-state",
		Arguments: (*Arguments)(self),
	})
}

type TraceEventInfo struct {
Name string json:"name"
State string json:"state"
}

func (self *TraceEventInfo) UnmarshalJSON(data []byte) error {
type Return TraceEventInfo
ret := struct {
Return Return json:"return"
}{}
err := json.Unmarshal(data, &ret)
if err != nil {
return err
}
self.Name = ret.Return.Name
self.State = ret.Return.State
return nil
}

func main() {
	var err error
	var qmp_input []byte
	var qmp_output []byte

	cmd := TraceEventGetStateCommand{
		Name: "qemu_memalign",
	}
	if qmp_input, err = json.Marshal(&cmd); err != nil {
		panic(err)
	}
	fmt.Printf("   cmd: %v\n", cmd)
	fmt.Printf("-> qmp_input: %v\n", string(qmp_input))

	qmp_output = ([]byte)(`{"return":{"name":"qemu_memalign","state":"disabled"}}`)
	ret := TraceEventInfo{}
	if err = json.Unmarshal(qmp_output, &ret); err != nil {
		panic(err)
	}
	fmt.Printf("<- qmp_output: %v\n", string(qmp_output))
	fmt.Printf("   ret: %v\n", ret)
}
#23 lore raw
Quoted 3 lines · Andrea Bolognani

Dealing with errors and commands that don't have a return value might require us to have generic CommandResult wrapper after all, but we should really try as hard as we can to stick to type safe interfaces.

On second thought, this wouldn't actually need to be generic: we could have something like

  type TraceEventGetStateResult struct {
      Result TraceEventInfo `json:"return"`
      Error  *Error         `json:"error"`
  }

and the caller would check that res.Error is nil before accessing res.Result.

Commands for which a return value is not expected would just have the Error part in their corresponding Result struct, and those that can either return an object or nothing (are there actually any like that?) could have a pointer as the Result member.

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
#24 lore raw
In reply to Andrea Bolognani
Quoted 8 lines · Andrea Bolognani
Quoted 2 lines · Victor Toso

Thanks for taking a look, let me know if you have questions, ideas or suggestions.

Full disclosure: I have only given the actual implementation a very cursory look so far, and I've focused on the generated Go API instead.

Quoted 6 lines

Overall things look pretty good.

One concern that I have is about naming struct members: things like SpiceInfo.MouseMode and most others are translated from the QAPI schema exactly the way you'd expect them, but for example ChardevCommon.Logappend doesn't look quite right.

It doesn't look quite right in the QAPI schema, either: @logappend. If it was @log-append, as it should, then it would get translated to LogAppend, I guess.

Fixing up style isn't a code generator's job.

Quoted 2 lines

Of course there's
no way to programmatically figure out what to capitalize,

Some case conversions are straightforward enough. For instance, the C code generator generates qapi_event_send_some_event() for event SOME_EVENT, and inclusion guard macro FILE_NAME_H for module file name file-name.json. No magic involved.

Conversion from lower-case-with-dashes to CamelCase doesn't have to be magic, either. You just have to accept garbage-in (like missing dashes) will give you garbage-out. I wouldn't care too much; CamelCaseIsAnIllegibleMessAnyway.

Conversion from CamelCase is always trouble, though. There is one instance so far: generating C enumeration constants. We want TYPE_NAME
+ '_' + MEMBER_NAME, where TYPE_NAME is the enumeration type's name converted from CamelCase to UPPER_CASE_WITH_UNDERSCORES, and MEMBER_NAME is the member name converted from lower-case-with-dashes to UPPER_CASE_WITH_UNDERSCORES.

camel_to_upper() tries, but the result is often unappealing, surprising, or both.

Quoted 3 lines

but maybe
there's room for adding this kind of information in the form of additional annotations or something like that?

We did for enumeration types: 'prefix' overrides the TYPE_NAME prefix. I fear this was a mistake.

Quoted 3 lines

Same for the various
structs or members that have unexpectedly-capitalized "Tls" or "Vnc" in them.

Examples?

Quoted 2 lines

To be clear, I don't think the above is a blocker - just something to be aware of, and think about.

Yup.

[...]

#25 lore raw
Quoted 11 lines · Markus Armbruster
Quoted 4 lines · Andrea Bolognani

One concern that I have is about naming struct members: things like SpiceInfo.MouseMode and most others are translated from the QAPI schema exactly the way you'd expect them, but for example ChardevCommon.Logappend doesn't look quite right.

It doesn't look quite right in the QAPI schema, either: @logappend. If it was @log-append, as it should, then it would get translated to LogAppend, I guess.

Fixing up style isn't a code generator's job.

I agree that the generator shouldn't take too many liberties when translating names, and specifically should never attempt to figure out that @logappend should have been @log-append instead.

What I was thinking of was more along the lines of, can we change the schema so that the proper name is available to the generator without breaking the wire protocol? Maybe something like

##
# ChardevCommon:
#
# @logappend (rename @log-append): ...
##

That way the generator would have access to both information, and would thus be able to generate

  type ChardevCommon struct {
    LogAppend *bool `json:"logappend,omitempty"`
  }

The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.

Quoted 5 lines
Quoted 3 lines

Same for the various
structs or members that have unexpectedly-capitalized "Tls" or "Vnc" in them.

Examples?

A perfect one is TlsCredsProperties, whose endpoint member is of type QCryptoTLSCredsEndpoint.

On the VNC front, we have SetPasswordOptionsVnc but also DisplayReloadOptionsVNC.

There's plenty more, but this should be illustrative enough already. Capitalization of these acronyms is inconsistent across the schema, with one of the two forms disagreeing with Go naming expectations.

In this case we might be able to just change the schema without introducing backwards compatibility issues, though? Type names are not actually transmitted on the wire IIUC.

Quoted 6 lines
Quoted 3 lines

but maybe
there's room for adding this kind of information in the form of additional annotations or something like that?

We did for enumeration types: 'prefix' overrides the TYPE_NAME prefix. I fear this was a mistake.

This might be an oversimplification, but I think we might be able to solve all of these issues with a single annotation in the form

namespace:word-MLA-other-words

So for example QCryptoTLSCredsEndpoint would be annotated with

q:crypto-TLS-creds-endpoint

and each generator would have enough information to produce identifiers that fit into the corresponding language, such as

qcrypto_tls_creds_endpoint
CryptoTlsCredsEndpoint

or whatever else.

Of course such annotations would only be necessary to deal with identifiers that are not already following the expected naming conventions and when MLAs are involved.

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
#26 lore raw
Quoted 30 lines · Andrea Bolognani
Quoted 11 lines · Markus Armbruster
Quoted 4 lines · Andrea Bolognani

One concern that I have is about naming struct members: things like SpiceInfo.MouseMode and most others are translated from the QAPI schema exactly the way you'd expect them, but for example ChardevCommon.Logappend doesn't look quite right.

It doesn't look quite right in the QAPI schema, either: @logappend. If it was @log-append, as it should, then it would get translated to LogAppend, I guess.

Fixing up style isn't a code generator's job.

I agree that the generator shouldn't take too many liberties when translating names, and specifically should never attempt to figure out that @logappend should have been @log-append instead.

What I was thinking of was more along the lines of, can we change the schema so that the proper name is available to the generator without breaking the wire protocol? Maybe something like

##
# ChardevCommon:
#
# @logappend (rename @log-append): ...
##

That way the generator would have access to both information, and would thus be able to generate

Quoted 6 lines
  type ChardevCommon struct {
    LogAppend *bool `json:"logappend,omitempty"`
  }

The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.

At the price of mild inconsistency between the library interface and QMP.

We could clean up QMP if we care, keeping around the old names for compatibility. See also Kevin's work on QAPI aliases. Which is much more ambitious, though.

Quoted 6 lines
Quoted 5 lines
Quoted 3 lines

Same for the various
structs or members that have unexpectedly-capitalized "Tls" or "Vnc" in them.

Examples?

Quoted 8 lines

A perfect one is TlsCredsProperties, whose endpoint member is of type QCryptoTLSCredsEndpoint.

On the VNC front, we have SetPasswordOptionsVnc but also DisplayReloadOptionsVNC.

There's plenty more, but this should be illustrative enough already. Capitalization of these acronyms is inconsistent across the schema,

Common issue with camel-cased compounds containing acronyms, because either way is ugly.

Quoted 1 line

with one of the two forms disagreeing with Go naming expectations.

Pardon my ignorance: What are Go's expectations?

Quoted 3 lines

In this case we might be able to just change the schema without introducing backwards compatibility issues, though? Type names are not actually transmitted on the wire IIUC.

Correct!

Quoted 19 lines
Quoted 6 lines
Quoted 3 lines

but maybe
there's room for adding this kind of information in the form of additional annotations or something like that?

We did for enumeration types: 'prefix' overrides the TYPE_NAME prefix. I fear this was a mistake.

This might be an oversimplification, but I think we might be able to solve all of these issues with a single annotation in the form

namespace:word-MLA-other-words

So for example QCryptoTLSCredsEndpoint would be annotated with

q:crypto-TLS-creds-endpoint

and each generator would have enough information to produce identifiers that fit into the corresponding language, such as

Quoted 8 lines

qcrypto_tls_creds_endpoint
CryptoTlsCredsEndpoint

or whatever else.

Of course such annotations would only be necessary to deal with identifiers that are not already following the expected naming conventions and when MLAs are involved.

Pardon my ignorance some more: what are MLAs?

#27 lore raw
Quoted 6 lines · Markus Armbruster
Quoted 2 lines · Andrea Bolognani

The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.

At the price of mild inconsistency between the library interface and QMP.

That's fine, and in fact it already happens all the time when QAPI names (log-append) are translated to C identifiers (log_append).

Quoted 3 lines

We could clean up QMP if we care, keeping around the old names for compatibility. See also Kevin's work on QAPI aliases. Which is much more ambitious, though.

I wasn't aware of that effort. Personally I'm always in favor of cleaning up inconsistencies, so I am automatically a fan :)

That said, the idea of exposing a sub-par Go API until such massive undertaking can be completed is not terribly appealing. And it would not address every facet of the issue (see below).

Quoted 4 lines
Quoted 1 line

Capitalization of these acronyms is inconsistent across the schema,

Common issue with camel-cased compounds containing acronyms, because either way is ugly.

Agreed :) But consistent ugliness is still preferable to inconsistent ugliness.

Quoted 3 lines
Quoted 1 line

with one of the two forms disagreeing with Go naming expectations.

Pardon my ignorance: What are Go's expectations?

Acronyms are usually all upper case:

https://pkg.go.dev/net/http#ParseHTTPVersion
https://pkg.go.dev/net/http#ProxyURL
https://pkg.go.dev/crypto/tls#NewLRUClientSessionCache

The same seems to be true of Python:

https://docs.python.org/3/library/http.html#http.HTTPStatus
https://docs.python.org/3/library/urllib.error.html#urllib.error.URLError
https://docs.python.org/3/library/xmlrpc.server.html#xmlrpc.server.SimpleXMLRPCServer

Rust, on the other hand, seems to prefer only capitalizing the first letter of a word, no matter if it's an acronym:

https://doc.rust-lang.org/std/net/struct.TcpStream.html
https://doc.rust-lang.org/std/net/struct.Ipv4Addr.html
https://doc.rust-lang.org/std/ffi/struct.OsString.html

Whether different naming conventions are used for types, functions and struct members is also language-dependent.

Quoted 5 lines
Quoted 3 lines

In this case we might be able to just change the schema without introducing backwards compatibility issues, though? Type names are not actually transmitted on the wire IIUC.

Correct!

That's great, but even if we decided to change all type names so that the schema is internally consistent and follows a naming convention that's reasonable for C, Go and Python, we'd still leave the Rust interface looking weird... There's no one-size-fits-all name, unfortunately.

Quoted 5 lines
Quoted 3 lines

Of course such annotations would only be necessary to deal with identifiers that are not already following the expected naming conventions and when MLAs are involved.

Pardon my ignorance some more: what are MLAs?

Multi Letter Acronyms. Which are actually just called "acronyms" I guess? O:-)

The point is that, if we want to provide a language interface that feels natural, we need a way to mark parts of a QAPI symbol's name in a way that makes it possible for the generator to know they're acronyms and treat them in an appropriate, language-specific manner.

The obvious way to implement this would be with an annotation along the lines of the one I proposed. Other ideas?

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
#28 lore raw
Quoted 10 lines · Andrea Bolognani
Quoted 6 lines · Markus Armbruster
Quoted 2 lines · Andrea Bolognani

The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.

At the price of mild inconsistency between the library interface and QMP.

That's fine, and in fact it already happens all the time when QAPI names (log-append) are translated to C identifiers (log_append).

There's a difference between trivial translations like "replace '-' by '_'" and arbitrary replacement like the one for enumeration constants involving 'prefix'.

Quoted 9 lines
Quoted 3 lines

We could clean up QMP if we care, keeping around the old names for compatibility. See also Kevin's work on QAPI aliases. Which is much more ambitious, though.

I wasn't aware of that effort. Personally I'm always in favor of cleaning up inconsistencies, so I am automatically a fan :)

That said, the idea of exposing a sub-par Go API until such massive undertaking can be completed is not terribly appealing.

Point.

Quoted 10 lines

And it would
not address every facet of the issue (see below).

Quoted 4 lines
Quoted 1 line

Capitalization of these acronyms is inconsistent across the schema,

Common issue with camel-cased compounds containing acronyms, because either way is ugly.

Agreed :) But consistent ugliness is still preferable to inconsistent ugliness.

True.

Quoted 16 lines
Quoted 3 lines
Quoted 1 line

with one of the two forms disagreeing with Go naming expectations.

Pardon my ignorance: What are Go's expectations?

Acronyms are usually all upper case:

https://pkg.go.dev/net/http#ParseHTTPVersion
https://pkg.go.dev/net/http#ProxyURL
https://pkg.go.dev/crypto/tls#NewLRUClientSessionCache

The same seems to be true of Python:

https://docs.python.org/3/library/http.html#http.HTTPStatus
https://docs.python.org/3/library/urllib.error.html#urllib.error.URLError
https://docs.python.org/3/library/xmlrpc.server.html#xmlrpc.server.SimpleXMLRPCServer

Quoted 6 lines

Rust, on the other hand, seems to prefer only capitalizing the first letter of a word, no matter if it's an acronym:

https://doc.rust-lang.org/std/net/struct.TcpStream.html
https://doc.rust-lang.org/std/net/struct.Ipv4Addr.html
https://doc.rust-lang.org/std/ffi/struct.OsString.html

Another strange game where the only winning move is not to play.

Quoted 2 lines

Whether different naming conventions are used for types, functions and struct members is also language-dependent.

Yes.

Quoted 11 lines
Quoted 5 lines
Quoted 3 lines

In this case we might be able to just change the schema without introducing backwards compatibility issues, though? Type names are not actually transmitted on the wire IIUC.

Correct!

That's great, but even if we decided to change all type names so that the schema is internally consistent and follows a naming convention that's reasonable for C, Go and Python, we'd still leave the Rust interface looking weird... There's no one-size-fits-all name, unfortunately.

C will remain the primary customer for quite a while. It doesn't come with universally accepted naming conventions, so we made up our own. I think we have some wiggle room there. We could, for instance, decide to clean up the current inconsistent capitalization of acronyms in the QAPI schema to either style, TCPStream or TcpStream.

Your point remains: some names will still look "weird" in some possible target languages.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Of course such annotations would only be necessary to deal with identifiers that are not already following the expected naming conventions and when MLAs are involved.

Pardon my ignorance some more: what are MLAs?

Multi Letter Acronyms. Which are actually just called "acronyms" I guess? O:-)

Well played!

Quoted 4 lines

The point is that, if we want to provide a language interface that feels natural, we need a way to mark parts of a QAPI symbol's name in a way that makes it possible for the generator to know they're acronyms and treat them in an appropriate, language-specific manner.

It's not just acronyms. Consider IAmALittleTeapot. If you can assume that only beginning of words are capitalized, even for acronyms, you can split this into words without trouble. You can't recover correct case,
though: "i am a little teapot" is wrong.

"Split before capital letter" falls apart when you have characters that cannot be capitalized: Point3d.

Camel case is hopeless.

Quoted 2 lines

The obvious way to implement this would be with an annotation along the lines of the one I proposed. Other ideas?

I'm afraid having the schema spell out names in multiple naming conventions could be onerous. How many names will need it? Times how many naming conventions?

Another issue: the fancier the translation from schema name to language-specific name gets, the harder it becomes to find one from the other.

#29 lore raw
Quoted 7 lines · Markus Armbruster
Quoted 6 lines · Andrea Bolognani
Quoted 5 lines
Quoted 2 lines

The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.

At the price of mild inconsistency between the library interface and QMP.

Quoted 6 lines
Quoted 2 lines

That's fine, and in fact it already happens all the time when QAPI names (log-append) are translated to C identifiers (log_append).

There's a difference between trivial translations like "replace '-' by '_'" and arbitrary replacement like the one for enumeration constants involving 'prefix'.

Fair enough.

I still feel that 1) users of a language SDK will ideally not need to look at the QAPI schema or wire chatter too often and 2) even when that ends up being necessary, figuring out that LogAppend and logappend are the same thing is not going to be an unreasonable hurdle, especially when the status quo would be to work with Logappend instead.

Quoted 9 lines
Quoted 4 lines

The point is that, if we want to provide a language interface that feels natural, we need a way to mark parts of a QAPI symbol's name in a way that makes it possible for the generator to know they're acronyms and treat them in an appropriate, language-specific manner.

It's not just acronyms. Consider IAmALittleTeapot. If you can assume that only beginning of words are capitalized, even for acronyms, you can split this into words without trouble. You can't recover correct case,
though: "i am a little teapot" is wrong.

Is there any scenario in which we would care though? We're in the business of translating identifiers from one machine representation to another, so once it has been split up correctly into the words that compose it (which in your example above it has) then we don't really care about anything else unless acronyms are involved.

In other words, we can obtain the list of words "i am a little teapot" programmatically both from IAmALittleTeapot and i-am-a-little-teapot, and in both cases we can then generate IAmALittleTeapot or I_AM_A_LITTLE_TEAPOT or i_am_a_little_teapot or whatever is appropriate for the context and target language, but the fact that in a proper English sentence "I" would have to be capitalized doesn't really enter the picture.

Quoted 4 lines

"Split before capital letter" falls apart when you have characters that cannot be capitalized: Point3d.

Camel case is hopeless.

I would argue that it works quite well for most scenarios, but there are some corner cases where it's clearly not good enough. If we can define a way to clue in the generator about "Point3d" having to be interpreted as "point 3d" and "VNCProps" as "vnc props", then we are golden. That wouldn't be necessary for simple cases that are already handled correctly.

A more radical idea would be to start using dash-notation for types too. That'd remove the word splitting issue altogether, at the cost of the schema being (possibly) harder to read and more distanced from the generated code.

You'd still only be able to generate VncProps from vnc-props though.

Quoted 5 lines
Quoted 2 lines

The obvious way to implement this would be with an annotation along the lines of the one I proposed. Other ideas?

I'm afraid having the schema spell out names in multiple naming conventions could be onerous. How many names will need it?

I don't have hard data on this. I could try extracting it, but that might end up being a bigger job than I had anticipated.

My guess is that the number of cases where the naive algorithm can't split words correctly is relatively small compared to the size of the entire QAPI schema. Fair warning: I have made incorrect guesses in the past ;)

Quoted 1 line

Times how many naming conventions?

Yeah, I don't think requiring all possible permutations to be spelled out in the schema is the way to go. That's exactly why my proposal was to offer a way to inject the semantic information that the parser can't figure out itself.

Once you have a way to inform the generator that "VNCProps" is made of the two words "vnc" and "props", and that "vnc" is an acronym, then it can generate an identifier appropriate for the target language without having to spell out anywhere that such an identifier would be VNCProps for Go and VncProps for Rust.

By the way, while looking around I realized that we also have to take into account things like D-Bus: the QAPI type ChardevDBus, for example, would probably translate verbatim to Go but have to be changed to ChardevDbus for Rust. Fun :)

Revised proposal for the annotation:

ns:word-WORD-WoRD-123Word

Words are always separated by dashes; "regular" words are entirely lowercase, while the presence of even a single uppercase letter in a word denotes the fact that its case should be preserved when the naming conventions of the target language allow that.

Quoted 3 lines

Another issue: the fancier the translation from schema name to language-specific name gets, the harder it becomes to find one from the other.

That's true, but at least to me the trade-off feels reasonable.

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
#30 lore raw
Quoted 11 lines · Andrea Bolognani
Quoted 10 lines · Markus Armbruster
Quoted 8 lines · Andrea Bolognani
Quoted 5 lines
Quoted 2 lines

The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.

At the price of mild inconsistency between the library interface and QMP.

That's fine, and in fact it already happens all the time when QAPI names (log-append) are translated to C identifiers (log_append).

Quoted 8 lines
Quoted 3 lines

There's a difference between trivial translations like "replace '-' by '_'" and arbitrary replacement like the one for enumeration constants involving 'prefix'.

Fair enough.

I still feel that 1) users of a language SDK will ideally not need to look at the QAPI schema or wire chatter too often

I think the most likely point of contact is the QEMU QMP Reference Manual.

Quoted 5 lines

even when
that ends up being necessary, figuring out that LogAppend and logappend are the same thing is not going to be an unreasonable hurdle, especially when the status quo would be to work with Logappend instead.

Yes, it's "mild inconsistency", hardly an unreasonable hurdle. I think it gets in the way mostly when searching documentation. Differences in case are mostly harmless, just use case-insensitive search. Use of '_' vs '-' would also be harmless (just do the replacement), if the use of '-' in the schema was consistent. Sadly, it's not, and that's already a perennial low-level annoyance.

My point is: a name override feature like the one you propose needs to be used with discipline and restraint. Adds to reviewers' mental load. Needs to be worth it. I'm not saying it isn't, I'm just pointing out a cost.

Quoted 16 lines
Quoted 9 lines
Quoted 4 lines

The point is that, if we want to provide a language interface that feels natural, we need a way to mark parts of a QAPI symbol's name in a way that makes it possible for the generator to know they're acronyms and treat them in an appropriate, language-specific manner.

It's not just acronyms. Consider IAmALittleTeapot. If you can assume that only beginning of words are capitalized, even for acronyms, you can split this into words without trouble. You can't recover correct case,
though: "i am a little teapot" is wrong.

Is there any scenario in which we would care though? We're in the business of translating identifiers from one machine representation to another, so once it has been split up correctly into the words that compose it (which in your example above it has) then we don't really care about anything else unless acronyms are involved.

Quoted 7 lines

In other words, we can obtain the list of words "i am a little teapot" programmatically both from IAmALittleTeapot and i-am-a-little-teapot, and in both cases we can then generate IAmALittleTeapot or I_AM_A_LITTLE_TEAPOT or i_am_a_little_teapot or whatever is appropriate for the context and target language, but the fact that in a proper English sentence "I" would have to be capitalized doesn't really enter the picture.

My point is that conversion from CamelCase has two sub-problems:
splitting words and recovering case. Splitting words is easy when exactly the beginning of words is capitalized. Recovering case is guesswork. Most English words are all lower case, but some start with a capital letter, and acronyms are all caps.

Wild idea: assume all lower case, but keep a list of exceptions.

Quoted 11 lines
Quoted 4 lines

"Split before capital letter" falls apart when you have characters that cannot be capitalized: Point3d.

Camel case is hopeless.

I would argue that it works quite well for most scenarios, but there are some corner cases where it's clearly not good enough. If we can define a way to clue in the generator about "Point3d" having to be interpreted as "point 3d" and "VNCProps" as "vnc props", then we are golden. That wouldn't be necessary for simple cases that are already handled correctly.

Hyphenization rules? *Cough* *cough*

Quoted 4 lines

A more radical idea would be to start using dash-notation for types too. That'd remove the word splitting issue altogether, at the cost of the schema being (possibly) harder to read and more distanced from the generated code.

Yes.

Quoted 10 lines

You'd still only be able to generate VncProps from vnc-props though.

Quoted 5 lines
Quoted 2 lines

The obvious way to implement this would be with an annotation along the lines of the one I proposed. Other ideas?

I'm afraid having the schema spell out names in multiple naming conventions could be onerous. How many names will need it?

I don't have hard data on this. I could try extracting it, but that might end up being a bigger job than I had anticipated.

I figure extracting is easier for me than for you. But let's have a closer look at the job at hand first.

The QAPI schema language uses three naming styles:

* lower-case-with-hyphens for command and member names

Many names use upper case and '_'. See pragma command-name-exceptions and member-name-exceptions.

Some (many?) names lack separators between words (example: logappend).

* UPPER_CASE_WITH_UNDERSCORE for event names

* CamelCase for type names

Capitalization of words is inconsistent in places (example: VncInfo vs. DisplayReloadOptionsVNC).

What style conversions will we need for Go? Any other conversions come to mind?

What problems do these conversions have?

Quoted 23 lines

My guess is that the number of cases where the naive algorithm can't split words correctly is relatively small compared to the size of the entire QAPI schema. Fair warning: I have made incorrect guesses in the past ;)

Quoted 1 line

Times how many naming conventions?

Yeah, I don't think requiring all possible permutations to be spelled out in the schema is the way to go. That's exactly why my proposal was to offer a way to inject the semantic information that the parser can't figure out itself.

Once you have a way to inform the generator that "VNCProps" is made of the two words "vnc" and "props", and that "vnc" is an acronym, then it can generate an identifier appropriate for the target language without having to spell out anywhere that such an identifier would be VNCProps for Go and VncProps for Rust.

By the way, while looking around I realized that we also have to take into account things like D-Bus: the QAPI type ChardevDBus, for example, would probably translate verbatim to Go but have to be changed to ChardevDbus for Rust. Fun :)

Quoted 8 lines

Revised proposal for the annotation:

ns:word-WORD-WoRD-123Word

Words are always separated by dashes; "regular" words are entirely lowercase, while the presence of even a single uppercase letter in a word denotes the fact that its case should be preserved when the naming conventions of the target language allow that.

Is a word always capitalized the same for a single target language? Or could capitalization depend on context?

Quoted 5 lines
Quoted 3 lines

Another issue: the fancier the translation from schema name to language-specific name gets, the harder it becomes to find one from the other.

That's true, but at least to me the trade-off feels reasonable.

#31 lore raw
Quoted 6 lines · Markus Armbruster
Quoted 2 lines · Andrea Bolognani

I still feel that 1) users of a language SDK will ideally not need to look at the QAPI schema or wire chatter too often

I think the most likely point of contact is the QEMU QMP Reference Manual.

Note that there isn't anything preventing us from including the original QAPI name in the documentation for the corresponding Go symbol, or even a link to the reference manual.

So we could make jumping from the Go API documentation, which is what a Go programmer will be looking at most of the time, to the QMP documentation pretty much effortless.

Quoted 4 lines

My point is: a name override feature like the one you propose needs to be used with discipline and restraint. Adds to reviewers' mental load. Needs to be worth it. I'm not saying it isn't, I'm just pointing out a cost.

Yeah, I get that.

Note that I'm not suggesting it should be possible for a name to be completely overridden - I just want to make it possible for a human to provide the name parsing algorithm solutions to those problems it can't figure out on its own.

We could prevent that feature from being misused by verifying that the symbol the annotation is attached to can be derived from the list of words provided. That way, something like

# SOMEName (completely-DIFFERENT-name)

would be rejected and we would avoid misuse.

Quoted 1 line

Wild idea: assume all lower case, but keep a list of exceptions.

That could actually work reasonably well for QEMU because we only need to handle correctly what's in the schema, not arbitrary input.

There's always the risk of the list of exceptions getting out of sync with the needs of the schema, but there's similarly no guarantee that annotations are going to be introduced when they are necessary, so it's mostly a wash.

The only slight advantage of the annotation approach would be that it might be easier to notice it being missing because it's close to the name it refers to, while the list of exceptions is tucked away in a script far away from it.

Quoted 6 lines

The QAPI schema language uses three naming styles:

* lower-case-with-hyphens for command and member names

Many names use upper case and '_'. See pragma command-name-exceptions and member-name-exceptions.

Looking at the output generated by Victor's WIP script, it looks like these are already handled as nicely as those that don't fall under any exception.

Quoted 6 lines

Some (many?) names lack separators between words (example: logappend).

* UPPER_CASE_WITH_UNDERSCORE for event names

* CamelCase for type names

Quoted 7 lines

Capitalization of words is inconsistent in places (example: VncInfo vs. DisplayReloadOptionsVNC).

What style conversions will we need for Go? Any other conversions come to mind?

What problems do these conversions have?

Go uses CamelCase for pretty much everything: types, methods, constants...

There's one slight wrinkle, in that the case of the first letter decides whether it's going to be a PublicName or a privateName. We can't do anything about that, but it shouldn't really affect us that much because we'll want all QAPI names to be public.

So the issues preventing us from producing a "perfect" Go API are

1. inconsistent capitalization in type names

-> could be addressed by simply changing the schema, as type
names do not travel on the wire

2. missing dashes in certain command/member names

-> leads to Incorrectcamelcase. Kevin's work is supposed to
address this

3. inability to know which parts of a lower-case-name or UPPER_CASE_NAME are acronyms or are otherwise supposed to be capitalized in a specific way

-> leads to WeirdVncAndDbusCapitalization. There's currently no
way, either implemented or planned, to avoid this

In addition to these I'm also thinking that QKeyCode and all the QCrypto stuff should probably lose their prefixes.

Note that 3 shouldn't be an issue for Rust and addressing 1 would actually make things worse for that language, because at the moment at least *some* of the types follow its expected naming rules :)

Quoted 11 lines
Quoted 8 lines

Revised proposal for the annotation:

ns:word-WORD-WoRD-123Word

Words are always separated by dashes; "regular" words are entirely lowercase, while the presence of even a single uppercase letter in a word denotes the fact that its case should be preserved when the naming conventions of the target language allow that.

Is a word always capitalized the same for a single target language? Or could capitalization depend on context?

I'm not aware of any language that would adopt more than a single style of capitalization, outside of course the obvious lower_case_name or UPPER_CASE_NAME scenarios where the original capitalization stops being relevant.

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
#32 lore raw

Am 03.05.2022 um 11:40 hat Andrea Bolognani geschrieben:

Quoted 11 lines

So the issues preventing us from producing a "perfect" Go API are

1. inconsistent capitalization in type names

-> could be addressed by simply changing the schema, as type
names do not travel on the wire

2. missing dashes in certain command/member names

-> leads to Incorrectcamelcase. Kevin's work is supposed to
address this

I am surprised that Markus pointed you to my aliases work because the conclusion after reviewing my last attempt was that he doesn't want the feature in the proposed form and he wasn't sure if he wants it at all. To be honest, I considered this one dead.

Even if magically a solution appeared that everyone agreed to, don't hold your breath, I'm working on different things now and not planning to get back to QAPI stuff before 7.1 (and preferably not before meeting Markus in person and discussing the design before I spend the time to produce another thing that will never be merged). I am hoping to get back to QAPI later, but I was mostly planning with continuing the QOM integration, not aliases.

Kevin

#33 lore raw
In reply to Andrea Bolognani
Quoted 100 lines · Andrea Bolognani
Quoted 6 lines · Markus Armbruster
Quoted 2 lines · Andrea Bolognani

I still feel that 1) users of a language SDK will ideally not need to look at the QAPI schema or wire chatter too often

I think the most likely point of contact is the QEMU QMP Reference Manual.

Note that there isn't anything preventing us from including the original QAPI name in the documentation for the corresponding Go symbol, or even a link to the reference manual.

So we could make jumping from the Go API documentation, which is what a Go programmer will be looking at most of the time, to the QMP documentation pretty much effortless.

Quoted 4 lines

My point is: a name override feature like the one you propose needs to be used with discipline and restraint. Adds to reviewers' mental load. Needs to be worth it. I'm not saying it isn't, I'm just pointing out a cost.

Yeah, I get that.

Note that I'm not suggesting it should be possible for a name to be completely overridden - I just want to make it possible for a human to provide the name parsing algorithm solutions to those problems it can't figure out on its own.

We could prevent that feature from being misused by verifying that the symbol the annotation is attached to can be derived from the list of words provided. That way, something like

# SOMEName (completely-DIFFERENT-name)

would be rejected and we would avoid misuse.

Quoted 1 line

Wild idea: assume all lower case, but keep a list of exceptions.

That could actually work reasonably well for QEMU because we only need to handle correctly what's in the schema, not arbitrary input.

There's always the risk of the list of exceptions getting out of sync with the needs of the schema, but there's similarly no guarantee that annotations are going to be introduced when they are necessary, so it's mostly a wash.

The only slight advantage of the annotation approach would be that it might be easier to notice it being missing because it's close to the name it refers to, while the list of exceptions is tucked away in a script far away from it.

Quoted 6 lines

The QAPI schema language uses three naming styles:

* lower-case-with-hyphens for command and member names

Many names use upper case and '_'. See pragma command-name-exceptions and member-name-exceptions.

Looking at the output generated by Victor's WIP script, it looks like these are already handled as nicely as those that don't fall under any exception.

Quoted 6 lines

Some (many?) names lack separators between words (example: logappend).

* UPPER_CASE_WITH_UNDERSCORE for event names

* CamelCase for type names

Quoted 7 lines

Capitalization of words is inconsistent in places (example: VncInfo vs. DisplayReloadOptionsVNC).

What style conversions will we need for Go? Any other conversions come to mind?

What problems do these conversions have?

Go uses CamelCase for pretty much everything: types, methods, constants...

There's one slight wrinkle, in that the case of the first letter decides whether it's going to be a PublicName or a privateName. We can't do anything about that, but it shouldn't really affect us that much because we'll want all QAPI names to be public.

So the issues preventing us from producing a "perfect" Go API are

1. inconsistent capitalization in type names

-> could be addressed by simply changing the schema, as type
names do not travel on the wire

2. missing dashes in certain command/member names

-> leads to Incorrectcamelcase. Kevin's work is supposed to
address this

3. inability to know which parts of a lower-case-name or UPPER_CASE_NAME are acronyms or are otherwise supposed to be capitalized in a specific way

Quoted 5 lines

-> leads to WeirdVncAndDbusCapitalization. There's currently no
way, either implemented or planned, to avoid this

In addition to these I'm also thinking that QKeyCode and all the QCrypto stuff should probably lose their prefixes.

At the C level, those prefixes are pretty important to avoid clashing with symbols defined by system headers and/or the external crypto library headers, as the unprefixed names are too generic. All non-C languages of course have proper namespacing

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Revised proposal for the annotation:

ns:word-WORD-WoRD-123Word

Words are always separated by dashes; "regular" words are entirely lowercase, while the presence of even a single uppercase letter in a word denotes the fact that its case should be preserved when the naming conventions of the target language allow that.

Quoted 7 lines
Quoted 2 lines

Is a word always capitalized the same for a single target language? Or could capitalization depend on context?

I'm not aware of any language that would adopt more than a single style of capitalization, outside of course the obvious lower_case_name or UPPER_CASE_NAME scenarios where the original capitalization stops being relevant.

As long as the capitalization we use in the schema can provides sufficient info to detect word splitting and acronyms, we can cope with any output language naming rules that I know of.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#34 lore raw
In reply to Andrea Bolognani
Quoted 28 lines · Andrea Bolognani
Quoted 6 lines · Markus Armbruster
Quoted 2 lines · Andrea Bolognani

I still feel that 1) users of a language SDK will ideally not need to look at the QAPI schema or wire chatter too often

I think the most likely point of contact is the QEMU QMP Reference Manual.

Note that there isn't anything preventing us from including the original QAPI name in the documentation for the corresponding Go symbol, or even a link to the reference manual.

So we could make jumping from the Go API documentation, which is what a Go programmer will be looking at most of the time, to the QMP documentation pretty much effortless.

Quoted 4 lines

My point is: a name override feature like the one you propose needs to be used with discipline and restraint. Adds to reviewers' mental load. Needs to be worth it. I'm not saying it isn't, I'm just pointing out a cost.

Yeah, I get that.

Note that I'm not suggesting it should be possible for a name to be completely overridden - I just want to make it possible for a human to provide the name parsing algorithm solutions to those problems it can't figure out on its own.

Quoted 7 lines

We could prevent that feature from being misused by verifying that the symbol the annotation is attached to can be derived from the list of words provided. That way, something like

# SOMEName (completely-DIFFERENT-name)

would be rejected and we would avoid misuse.

Possibly as simple as "down-case both names and drop the funny characters, result must be the same".

Quoted 10 lines
Quoted 1 line

Wild idea: assume all lower case, but keep a list of exceptions.

That could actually work reasonably well for QEMU because we only need to handle correctly what's in the schema, not arbitrary input.

There's always the risk of the list of exceptions getting out of sync with the needs of the schema, but there's similarly no guarantee that annotations are going to be introduced when they are necessary, so it's mostly a wash.

Quoted 4 lines

The only slight advantage of the annotation approach would be that it might be easier to notice it being missing because it's close to the name it refers to, while the list of exceptions is tucked away in a script far away from it.

We'd put it in qapi/pragma.json, I guess.

Quoted 12 lines
Quoted 6 lines

The QAPI schema language uses three naming styles:

* lower-case-with-hyphens for command and member names

Many names use upper case and '_'. See pragma command-name-exceptions and member-name-exceptions.

Looking at the output generated by Victor's WIP script, it looks like these are already handled as nicely as those that don't fall under any exception.

Quoted 1 line

Some (many?) names lack separators between words (example: logappend).

How many would be good to know.

Ad hoc hackery to find names, filter out camels (because word splitting is too hard there), split into words, look up words in a word list:

    $ for i in `/usr/bin/python3 /work/armbru/qemu/scripts/qapi-gen.py -o qapi -b ../qapi/qapi-schema.json | sort -u | awk '/^### [a-z0-9-]+$/ { print "lc", $2; next } /^### [a-z0-9_-]+$/ { print lu; next } /^### [A-Z0-9_]+$/ { print "uc", $2; next } /^### ([A-Z][a-z]+)+/ { print "cc", $2; next } { print "mc", $2 }' | sed '/^mc\|^cc/d;s/^.. //;s/[^A-Za-z0-9]/\n/g' | tr A-Z a-z | sort -u`; do grep -q "^$i$" /usr/share/dict/words || echo $i; done

420 lines. How many arguably lack separators between words? Wild guess based on glancing at the output sideways: some 50.

Quoted 20 lines
Quoted 11 lines

* UPPER_CASE_WITH_UNDERSCORE for event names

* CamelCase for type names

Capitalization of words is inconsistent in places (example: VncInfo vs. DisplayReloadOptionsVNC).

What style conversions will we need for Go? Any other conversions come to mind?

What problems do these conversions have?

Go uses CamelCase for pretty much everything: types, methods, constants...

There's one slight wrinkle, in that the case of the first letter decides whether it's going to be a PublicName or a privateName. We can't do anything about that, but it shouldn't really affect us that much because we'll want all QAPI names to be public.

Quoted 6 lines

So the issues preventing us from producing a "perfect" Go API are

1. inconsistent capitalization in type names

-> could be addressed by simply changing the schema, as type
names do not travel on the wire

At the price of some churn in C code.

Perhaps more consistent capitalization could be regarded as a slight improvement on its own. We need to see (a good sample of) the changes to judge.

Quoted 3 lines

2. missing dashes in certain command/member names

   -> leads to Incorrectcamelcase.

Names with words run together are arguably no uglier in CamelCase (Go) than in lower_case_with_underscores (C).

Quoted 2 lines

Kevin's work is supposed to
address this

Except it's stuck.

Perhaps Kevin and I can get it moving again.

Perhaps we can try to extract a local alias feature that can be grown into the more ambitious aliases Kevin wants (if we can solve the issues).

Quoted 6 lines

3. inability to know which parts of a lower-case-name or UPPER_CASE_NAME are acronyms or are otherwise supposed to be capitalized in a specific way

-> leads to WeirdVncAndDbusCapitalization. There's currently no
way, either implemented or planned, to avoid this

A list of words with special capitalization needs[*]?

VNC is an acronym, some languagues want VNC in camels, some Vnc.

DBus is an abbreviation, some languages want DBus in camels, some Dbus.

Quoted 2 lines

In addition to these I'm also thinking that QKeyCode and all the QCrypto stuff should probably lose their prefixes.

As Daniel pointed out, schema names sometimes have prefixes because we need the generated C identifiers to have prefixes.

If we hate these prefixes enough, we can try to limit them to C identifiers.

Quoted 3 lines

Note that 3 shouldn't be an issue for Rust and addressing 1 would actually make things worse for that language, because at the moment at least *some* of the types follow its expected naming rules :)

Solving Go problems by creating Rust problems doesn't feel like a good move to me.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Revised proposal for the annotation:

ns:word-WORD-WoRD-123Word

Words are always separated by dashes; "regular" words are entirely lowercase, while the presence of even a single uppercase letter in a word denotes the fact that its case should be preserved when the naming conventions of the target language allow that.

Quoted 7 lines
Quoted 2 lines

Is a word always capitalized the same for a single target language? Or could capitalization depend on context?

I'm not aware of any language that would adopt more than a single style of capitalization, outside of course the obvious lower_case_name or UPPER_CASE_NAME scenarios where the original capitalization stops being relevant.

Makes sense.

[*] Sounds like crony capitalism, doesn't it :)

#35 lore raw
In reply to Andrea Bolognani

Hi,

Quoted 13 lines · Andrea Bolognani
Quoted 12 lines · Markus Armbruster
Quoted 10 lines · Andrea Bolognani
Quoted 6 lines
Quoted 3 lines

The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.

At the price of mild inconsistency between the library interface and QMP.

That's fine, and in fact it already happens all the time when QAPI names (log-append) are translated to C identifiers (log_append).

Quoted 8 lines
Quoted 3 lines

There's a difference between trivial translations like "replace '-' by '_'" and arbitrary replacement like the one for enumeration constants involving 'prefix'.

Fair enough.

I still feel that 1) users of a language SDK will ideally not need to look at the QAPI schema or wire chatter too often and

That should be the preference, yes.

Quoted 4 lines

2) even when that ends up being necessary, figuring out that LogAppend and logappend are the same thing is not going to be an unreasonable hurdle, especially when the status quo would be to work with Logappend instead.

If user really needs to leave their ecosystem in order to check an alias, I hope we are already considering this a corner case. Still, if we are thinking about multiple languages communicating with QEMU, it is reasonable to consider the necessity of some docs page where they can easily grep/search for the types, alias, examples, etc. IMHO, this should be a long term goal and not a blocker... I can volunteer on working on that later.

Quoted 52 lines
Quoted 11 lines
Quoted 5 lines

The point is that, if we want to provide a language interface that feels natural, we need a way to mark parts of a QAPI symbol's name in a way that makes it possible for the generator to know they're acronyms and treat them in an appropriate, language-specific manner.

It's not just acronyms. Consider IAmALittleTeapot. If you can assume that only beginning of words are capitalized, even for acronyms, you can split this into words without trouble. You can't recover correct case, though: "i am a little teapot" is wrong.

Is there any scenario in which we would care though? We're in the business of translating identifiers from one machine representation to another, so once it has been split up correctly into the words that compose it (which in your example above it has) then we don't really care about anything else unless acronyms are involved.

In other words, we can obtain the list of words "i am a little teapot" programmatically both from IAmALittleTeapot and i-am-a-little-teapot, and in both cases we can then generate IAmALittleTeapot or I_AM_A_LITTLE_TEAPOT or i_am_a_little_teapot or whatever is appropriate for the context and target language, but the fact that in a proper English sentence "I" would have to be capitalized doesn't really enter the picture.

Quoted 4 lines

"Split before capital letter" falls apart when you have characters that cannot be capitalized: Point3d.

Camel case is hopeless.

I would argue that it works quite well for most scenarios, but there are some corner cases where it's clearly not good enough. If we can define a way to clue in the generator about "Point3d" having to be interpreted as "point 3d" and "VNCProps" as "vnc props", then we are golden. That wouldn't be necessary for simple cases that are already handled correctly.

A more radical idea would be to start using dash-notation for types too. That'd remove the word splitting issue altogether, at the cost of the schema being (possibly) harder to read and more distanced from the generated code.

You'd still only be able to generate VncProps from vnc-props though.

Quoted 4 lines
Quoted 3 lines

The obvious way to implement this would be with an annotation along the lines of the one I proposed. Other ideas?

Quoted 6 lines
Quoted 3 lines

I'm afraid having the schema spell out names in multiple naming conventions could be onerous. How many names will need it?

I don't have hard data on this. I could try extracting it, but that might end up being a bigger job than I had anticipated.

The only way to know is by checking /o\
I'll give it a shot.

Quoted 18 lines

My guess is that the number of cases where the naive algorithm can't split words correctly is relatively small compared to the size of the entire QAPI schema. Fair warning: I have made incorrect guesses in the past ;)

Quoted 1 line

Times how many naming conventions?

Yeah, I don't think requiring all possible permutations to be spelled out in the schema is the way to go. That's exactly why my proposal was to offer a way to inject the semantic information that the parser can't figure out itself.

Once you have a way to inform the generator that "VNCProps" is made of the two words "vnc" and "props", and that "vnc" is an acronym, then it can generate an identifier appropriate for the target language without having to spell out anywhere that such an identifier would be VNCProps for Go and VncProps for Rust.

Quoted 8 lines

By the way, while looking around I realized that we also have to take into account things like D-Bus: the QAPI type ChardevDBus, for example, would probably translate verbatim to Go but have to be changed to ChardevDbus for Rust. Fun :)

Revised proposal for the annotation:

ns:word-WORD-WoRD-123Word

I really like it.

Quoted 11 lines

Words are always separated by dashes; "regular" words are entirely lowercase, while the presence of even a single uppercase letter in a word denotes the fact that its case should be preserved when the naming conventions of the target language allow that.

Quoted 3 lines

Another issue: the fancier the translation from schema name to language-specific name gets, the harder it becomes to find one from the other.

That's true, but at least to me the trade-off feels reasonable.

I don't quite get the argument why it gets harder to find. We can simply provide the actual name as reference in the generated documentation, no?

As Kevin already pointed out that he is not planning to work on the Alias (reference below), any other idea besides the Andrea's annotation suggestion? While this is not a blocker, I agree it would be nice to be consistent.

    https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg04703.html

Cheers,
Victor

#36 lore raw
Quoted 96 lines · Victor Toso

Hi,

Quoted 13 lines · Andrea Bolognani
Quoted 12 lines · Markus Armbruster
Quoted 10 lines · Andrea Bolognani
Quoted 6 lines
Quoted 3 lines

The wire protocol would still retain the unappealing name, but at least client libraries could hide the uglyness from users.

At the price of mild inconsistency between the library interface and QMP.

That's fine, and in fact it already happens all the time when QAPI names (log-append) are translated to C identifiers (log_append).

Quoted 8 lines
Quoted 3 lines

There's a difference between trivial translations like "replace '-' by '_'" and arbitrary replacement like the one for enumeration constants involving 'prefix'.

Fair enough.

I still feel that 1) users of a language SDK will ideally not need to look at the QAPI schema or wire chatter too often and

That should be the preference, yes.

Quoted 4 lines

2) even when that ends up being necessary, figuring out that LogAppend and logappend are the same thing is not going to be an unreasonable hurdle, especially when the status quo would be to work with Logappend instead.

If user really needs to leave their ecosystem in order to check an alias, I hope we are already considering this a corner case. Still, if we are thinking about multiple languages communicating with QEMU, it is reasonable to consider the necessity of some docs page where they can easily grep/search for the types, alias, examples, etc. IMHO, this should be a long term goal and not a blocker... I can volunteer on working on that later.

Quoted 56 lines
Quoted 11 lines
Quoted 5 lines

The point is that, if we want to provide a language interface that feels natural, we need a way to mark parts of a QAPI symbol's name in a way that makes it possible for the generator to know they're acronyms and treat them in an appropriate, language-specific manner.

It's not just acronyms. Consider IAmALittleTeapot. If you can assume that only beginning of words are capitalized, even for acronyms, you can split this into words without trouble. You can't recover correct case, though: "i am a little teapot" is wrong.

Is there any scenario in which we would care though? We're in the business of translating identifiers from one machine representation to another, so once it has been split up correctly into the words that compose it (which in your example above it has) then we don't really care about anything else unless acronyms are involved.

In other words, we can obtain the list of words "i am a little teapot" programmatically both from IAmALittleTeapot and i-am-a-little-teapot, and in both cases we can then generate IAmALittleTeapot or I_AM_A_LITTLE_TEAPOT or i_am_a_little_teapot or whatever is appropriate for the context and target language, but the fact that in a proper English sentence "I" would have to be capitalized doesn't really enter the picture.

Quoted 4 lines

"Split before capital letter" falls apart when you have characters that cannot be capitalized: Point3d.

Camel case is hopeless.

I would argue that it works quite well for most scenarios, but there are some corner cases where it's clearly not good enough. If we can define a way to clue in the generator about "Point3d" having to be interpreted as "point 3d" and "VNCProps" as "vnc props", then we are golden. That wouldn't be necessary for simple cases that are already handled correctly.

A more radical idea would be to start using dash-notation for types too. That'd remove the word splitting issue altogether, at the cost of the schema being (possibly) harder to read and more distanced from the generated code.

You'd still only be able to generate VncProps from vnc-props though.

Quoted 7 lines
Quoted 3 lines

The obvious way to implement this would be with an annotation along the lines of the one I proposed. Other ideas?

I'm afraid having the schema spell out names in multiple naming conventions could be onerous. How many names will need it?

Quoted 5 lines
Quoted 2 lines

I don't have hard data on this. I could try extracting it, but that might end up being a bigger job than I had anticipated.

The only way to know is by checking /o\
I'll give it a shot.

I append a quick hack to find names in a QAPI schema, and sort them into buckets "lc" (lower-with-hyphen), "uc" (UPPER_WITH_UNDERSCORE), "cc" (CamelCase where words are obvious), and "mc" (everything else, mostly CamelCase where words aren't obvious).

Note that this ignores naming rule violations such as upper case enum member EAX.

Quoted 39 lines
Quoted 18 lines

My guess is that the number of cases where the naive algorithm can't split words correctly is relatively small compared to the size of the entire QAPI schema. Fair warning: I have made incorrect guesses in the past ;)

Quoted 1 line

Times how many naming conventions?

Yeah, I don't think requiring all possible permutations to be spelled out in the schema is the way to go. That's exactly why my proposal was to offer a way to inject the semantic information that the parser can't figure out itself.

Once you have a way to inform the generator that "VNCProps" is made of the two words "vnc" and "props", and that "vnc" is an acronym, then it can generate an identifier appropriate for the target language without having to spell out anywhere that such an identifier would be VNCProps for Go and VncProps for Rust.

Quoted 8 lines

By the way, while looking around I realized that we also have to take into account things like D-Bus: the QAPI type ChardevDBus, for example, would probably translate verbatim to Go but have to be changed to ChardevDbus for Rust. Fun :)

Revised proposal for the annotation:

ns:word-WORD-WoRD-123Word

I really like it.

Quoted 10 lines

Words are always separated by dashes; "regular" words are entirely lowercase, while the presence of even a single uppercase letter in a word denotes the fact that its case should be preserved when the naming conventions of the target language allow that.

Quoted 3 lines

Another issue: the fancier the translation from schema name to language-specific name gets, the harder it becomes to find one from the other.

Quoted 5 lines
Quoted 1 line

That's true, but at least to me the trade-off feels reasonable.

I don't quite get the argument why it gets harder to find. We can simply provide the actual name as reference in the generated documentation, no?

Predictable name transformation can save me detours through documentation. Being able to guess the Go bindings just from the QMP Reference Manual can be nice: it lets me write Go code with just the QMP Reference manual in view. Being able to guess the name in the QAPI schema from the name in the Go bindings can also be nice: it lets me look it up in the QMP Reference manual without jumping through the bindings documentation.

Or do you envisage a Go bindings manual that fully replaces the QMP Reference Manual for developers writing Go?

Quoted 6 lines

As Kevin already pointed out that he is not planning to work on the Alias (reference below), any other idea besides the Andrea's annotation suggestion? While this is not a blocker, I agree it would be nice to be consistent.

    https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg04703.html

Let's first try to get a handle on how many schema names are problematic.

$ git-diff
scripts/qapi/expr.py +1 -0
index 5a1782b57e..804b8ab455 100644
@@ -94,6 +94,7 @@ def check_name_str(name: str, info: QAPISourceInfo, source: str) -> str:
     """
     # Reserve the entire 'q_' namespace for c_name(), and for 'q_empty'
     # and 'q_obj_*' implicit type names.
+    print("###", name)
     match = valid_name.match(name)
     if not match or c_name(name, False).startswith('q_'):
         raise QAPISemError(info, "%s has an invalid name" % source)
$ cd bld
$ python3 ../scripts/qapi-gen.py -o qapi -b ../qapi/qapi-schema.json | sort -u | awk '/^### [a-z0-9-]+$/ { print "lc", $2; next } /^### [A-Z0-9_]+$/ { print "uc", $2; next } /^### ([A-Z][a-z]+)+/ { print "cc", $2; next } { print "mc", $2 }' | sort
cc Abort
cc AbortWrapper
cc AcpiTableOptions
cc ActionCompletionMode
cc AddfdInfo
cc AnnounceParameters
cc AudioFormat
cc Audiodev
cc AudiodevAlsaOptions
cc AudiodevAlsaPerDirectionOptions
cc AudiodevCoreaudioOptions
cc AudiodevCoreaudioPerDirectionOptions
cc AudiodevDriver
cc AudiodevDsoundOptions
cc AudiodevGenericOptions
cc AudiodevJackOptions
cc AudiodevJackPerDirectionOptions
cc AudiodevOssOptions
cc AudiodevOssPerDirectionOptions
cc AudiodevPaOptions
cc AudiodevPaPerDirectionOptions
cc AudiodevPerDirectionOptions
cc AudiodevSdlOptions
cc AudiodevSdlPerDirectionOptions
cc AudiodevWavOptions
cc AuthZListFileProperties
cc AuthZListProperties
cc AuthZPAMProperties
cc AuthZSimpleProperties
cc BackupCommon
cc BackupPerf
cc BalloonInfo
cc BiosAtaTranslation
cc BitmapMigrationBitmapAlias
cc BitmapMigrationBitmapAliasTransform
cc BitmapMigrationNodeAlias
cc BitmapSyncMode
cc BlkdebugEvent
cc BlkdebugIOType
cc BlkdebugInjectErrorOptions
cc BlkdebugSetStateOptions
cc BlockDeviceInfo
cc BlockDeviceIoStatus
cc BlockDeviceStats
cc BlockDeviceTimedStats
cc BlockDirtyBitmap
cc BlockDirtyBitmapAdd
cc BlockDirtyBitmapAddWrapper
cc BlockDirtyBitmapMerge
cc BlockDirtyBitmapMergeWrapper
cc BlockDirtyBitmapOrStr
cc BlockDirtyBitmapSha256
cc BlockDirtyBitmapWrapper
cc BlockDirtyInfo
cc BlockErrorAction
cc BlockExportInfo
cc BlockExportOptions
cc BlockExportOptionsFuse
cc BlockExportOptionsNbd
cc BlockExportOptionsNbdBase
cc BlockExportOptionsVhostUserBlk
cc BlockExportRemoveMode
cc BlockExportType
cc BlockIOThrottle
cc BlockInfo
cc BlockJobInfo
cc BlockLatencyHistogramInfo
cc BlockMeasureInfo
cc BlockPermission
cc BlockStats
cc BlockStatsSpecific
cc BlockStatsSpecificFile
cc BlockStatsSpecificNvme
cc BlockdevAioOptions
cc BlockdevAmendOptions
cc BlockdevAmendOptionsLUKS
cc BlockdevAmendOptionsQcow2
cc BlockdevBackup
cc BlockdevBackupWrapper
cc BlockdevCacheInfo
cc BlockdevCacheOptions
cc BlockdevChangeReadOnlyMode
cc BlockdevCreateOptions
cc BlockdevCreateOptionsFile
cc BlockdevCreateOptionsGluster
cc BlockdevCreateOptionsLUKS
cc BlockdevCreateOptionsNfs
cc BlockdevCreateOptionsParallels
cc BlockdevCreateOptionsQcow
cc BlockdevCreateOptionsQcow2
cc BlockdevCreateOptionsQed
cc BlockdevCreateOptionsRbd
cc BlockdevCreateOptionsSsh
cc BlockdevCreateOptionsVdi
cc BlockdevCreateOptionsVhdx
cc BlockdevCreateOptionsVmdk
cc BlockdevCreateOptionsVpc
cc BlockdevDetectZeroesOptions
cc BlockdevDiscardOptions
cc BlockdevDriver
cc BlockdevOnError
cc BlockdevOptions
cc BlockdevOptionsBlkdebug
cc BlockdevOptionsBlklogwrites
cc BlockdevOptionsBlkreplay
cc BlockdevOptionsBlkverify
cc BlockdevOptionsCbw
cc BlockdevOptionsCor
cc BlockdevOptionsCurlBase
cc BlockdevOptionsCurlFtp
cc BlockdevOptionsCurlFtps
cc BlockdevOptionsCurlHttp
cc BlockdevOptionsCurlHttps
cc BlockdevOptionsFile
cc BlockdevOptionsGenericCOWFormat
cc BlockdevOptionsGenericFormat
cc BlockdevOptionsGluster
cc BlockdevOptionsIscsi
cc BlockdevOptionsLUKS
cc BlockdevOptionsNVMe
cc BlockdevOptionsNbd
cc BlockdevOptionsNfs
cc BlockdevOptionsNull
cc BlockdevOptionsPreallocate
cc BlockdevOptionsQcow
cc BlockdevOptionsQcow2
cc BlockdevOptionsQuorum
cc BlockdevOptionsRaw
cc BlockdevOptionsRbd
cc BlockdevOptionsReplication
cc BlockdevOptionsSsh
cc BlockdevOptionsThrottle
cc BlockdevOptionsVVFAT
cc BlockdevQcow2Encryption
cc BlockdevQcow2EncryptionFormat
cc BlockdevQcow2Version
cc BlockdevQcowEncryption
cc BlockdevQcowEncryptionFormat
cc BlockdevRef
cc BlockdevRefOrNull
cc BlockdevSnapshot
cc BlockdevSnapshotInternal
cc BlockdevSnapshotInternalWrapper
cc BlockdevSnapshotSync
cc BlockdevSnapshotSyncWrapper
cc BlockdevSnapshotWrapper
cc BlockdevVhdxSubformat
cc BlockdevVmdkAdapterType
cc BlockdevVmdkSubformat
cc BlockdevVpcSubformat
cc CanHostSocketcanProperties
cc ChardevBackend
cc ChardevBackendInfo
cc ChardevBackendKind
cc ChardevCommon
cc ChardevCommonWrapper
cc ChardevDBus
cc ChardevDBusWrapper
cc ChardevFile
cc ChardevFileWrapper
cc ChardevHostdev
cc ChardevHostdevWrapper
cc ChardevInfo
cc ChardevMux
cc ChardevMuxWrapper
cc ChardevQemuVDAgent
cc ChardevQemuVDAgentWrapper
cc ChardevReturn
cc ChardevRingbuf
cc ChardevRingbufWrapper
cc ChardevSocket
cc ChardevSocketWrapper
cc ChardevSpiceChannel
cc ChardevSpiceChannelWrapper
cc ChardevSpicePort
cc ChardevSpicePortWrapper
cc ChardevStdio
cc ChardevStdioWrapper
cc ChardevUdp
cc ChardevUdpWrapper
cc ChardevVC
cc ChardevVCWrapper
cc ColoCompareProperties
cc CommandInfo
cc CommandLineOptionInfo
cc CommandLineParameterInfo
cc CommandLineParameterType
cc CommandNotFound
cc CompatPolicy
cc CompatPolicyInput
cc CompatPolicyOutput
cc CompressionStats
cc CpuDefinitionInfo
cc CpuInfoFast
cc CpuInfoS390
cc CpuInstanceProperties
cc CpuModelBaselineInfo
cc CpuModelCompareInfo
cc CpuModelCompareResult
cc CpuModelExpansionInfo
cc CpuModelExpansionType
cc CpuModelInfo
cc CpuS390State
cc CryptodevBackendProperties
cc CryptodevVhostUserProperties
cc CurrentMachineParams
cc DataFormat
cc DeviceNotActive
cc DeviceNotFound
cc DirtyRateInfo
cc DirtyRateMeasureMode
cc DirtyRateStatus
cc DirtyRateVcpu
cc DisplayCocoa
cc DisplayCurses
cc DisplayDBus
cc DisplayEGLHeadless
cc DisplayGLMode
cc DisplayGTK
cc DisplayOptions
cc DisplayProtocol
cc DisplayReloadOptions
cc DisplayReloadOptionsVNC
cc DisplayReloadType
cc DisplayType
cc DisplayUpdateOptions
cc DisplayUpdateOptionsVNC
cc DisplayUpdateType
cc DriveBackup
cc DriveBackupWrapper
cc DriveMirror
cc DummyForceArrays
cc DumpGuestMemoryCapability
cc DumpGuestMemoryFormat
cc DumpQueryResult
cc DumpStatus
cc EventLoopBaseProperties
cc ExpirePasswordOptions
cc ExpirePasswordOptionsVnc
cc FailoverStatus
cc FdsetFdInfo
cc FdsetInfo
cc FilterBufferProperties
cc FilterDumpProperties
cc FilterMirrorProperties
cc FilterRedirectorProperties
cc FilterRewriterProperties
cc FloppyDriveType
cc FuseExportAllowOther
cc GenericError
cc GrabToggleKeys
cc GuestPanicAction
cc GuestPanicInformation
cc GuestPanicInformationHyperV
cc GuestPanicInformationS390
cc GuestPanicInformationType
cc GuidInfo
cc HmatCacheAssociativity
cc HmatCacheWritePolicy
cc HmatLBDataType
cc HmatLBMemoryHierarchy
cc HostMemPolicy
cc HotpluggableCPU
cc HumanReadableText
cc ImageCheck
cc ImageFormat
cc ImageInfo
cc ImageInfoSpecific
cc ImageInfoSpecificKind
cc ImageInfoSpecificLUKSWrapper
cc ImageInfoSpecificQCow2
cc ImageInfoSpecificQCow2Encryption
cc ImageInfoSpecificQCow2EncryptionBase
cc ImageInfoSpecificQCow2Wrapper
cc ImageInfoSpecificRbd
cc ImageInfoSpecificRbdWrapper
cc ImageInfoSpecificVmdk
cc ImageInfoSpecificVmdkWrapper
cc InetSocketAddress
cc InetSocketAddressBase
cc InetSocketAddressWrapper
cc InputAxis
cc InputBarrierProperties
cc InputBtnEvent
cc InputBtnEventWrapper
cc InputButton
cc InputEvent
cc InputEventKind
cc InputKeyEvent
cc InputKeyEventWrapper
cc InputLinuxProperties
cc InputMoveEvent
cc InputMoveEventWrapper
cc IntWrapper
cc IoOperationType
cc IothreadProperties
cc IscsiHeaderDigest
cc IscsiTransport
cc JobInfo
cc JobStatus
cc JobType
cc JobVerb
cc KeyValue
cc KeyValueKind
cc KvmInfo
cc LostTickPolicy
cc MachineInfo
cc MainLoopProperties
cc MapEntry
cc Memdev
cc MemoryBackendEpcProperties
cc MemoryBackendFileProperties
cc MemoryBackendMemfdProperties
cc MemoryBackendProperties
cc MemoryDeviceInfo
cc MemoryDeviceInfoKind
cc MemoryFailureAction
cc MemoryFailureFlags
cc MemoryFailureRecipient
cc MemoryInfo
cc MigrateSetParameters
cc MigrationCapability
cc MigrationCapabilityStatus
cc MigrationInfo
cc MigrationParameter
cc MigrationParameters
cc MigrationStats
cc MigrationStatus
cc MirrorCopyMode
cc MirrorSyncMode
cc MonitorMode
cc MonitorOptions
cc MouseInfo
cc MultiFDCompression
cc NameInfo
cc NbdServerAddOptions
cc NbdServerOptions
cc NetClientDriver
cc NetFilterDirection
cc NetLegacyNicOptions
cc Netdev
cc NetdevBridgeOptions
cc NetdevHubPortOptions
cc NetdevL2TPv3Options
cc NetdevNetmapOptions
cc NetdevSocketOptions
cc NetdevTapOptions
cc NetdevUserOptions
cc NetdevVdeOptions
cc NetdevVhostUserOptions
cc NetdevVhostVDPAOptions
cc NetfilterInsert
cc NetfilterProperties
cc NetworkAddressFamily
cc NewImageMode
cc NumaCpuOptions
cc NumaDistOptions
cc NumaHmatCacheOptions
cc NumaHmatLBOptions
cc NumaNodeOptions
cc NumaOptions
cc NumaOptionsType
cc ObjectOptions
cc ObjectPropertyInfo
cc ObjectType
cc ObjectTypeInfo
cc OffAutoPCIBAR
cc OnOffAuto
cc OnOffSplit
cc PanicAction
cc PciBridgeInfo
cc PciBusInfo
cc PciDeviceClass
cc PciDeviceId
cc PciDeviceInfo
cc PciInfo
cc PciMemoryRange
cc PciMemoryRegion
cc PrManagerHelperProperties
cc PreallocMode
cc QapiErrorClass
cc Qcow2BitmapInfo
cc Qcow2BitmapInfoFlags
cc Qcow2CompressionType
cc Qcow2OverlapCheckFlags
cc Qcow2OverlapCheckMode
cc Qcow2OverlapChecks
cc QtestProperties
cc QuorumOpType
cc QuorumReadPattern
cc RbdAuthMode
cc RbdEncryptionCreateOptions
cc RbdEncryptionCreateOptionsLUKS
cc RbdEncryptionCreateOptionsLUKS2
cc RbdEncryptionCreateOptionsLUKSBase
cc RbdEncryptionOptions
cc RbdEncryptionOptionsLUKS
cc RbdEncryptionOptionsLUKS2
cc RbdEncryptionOptionsLUKSBase
cc RbdImageEncryptionFormat
cc RebootAction
cc RemoteObjectProperties
cc ReplayInfo
cc ReplayMode
cc ReplicationMode
cc ReplicationStatus
cc RngEgdProperties
cc RngProperties
cc RngRandomProperties
cc RockerOfDpaFlow
cc RockerOfDpaFlowAction
cc RockerOfDpaFlowKey
cc RockerOfDpaFlowMask
cc RockerOfDpaGroup
cc RockerPort
cc RockerPortAutoneg
cc RockerPortDuplex
cc RockerSwitch
cc RunState
cc RxFilterInfo
cc RxState
cc SchemaInfo
cc SchemaInfoAlternate
cc SchemaInfoAlternateMember
cc SchemaInfoArray
cc SchemaInfoBuiltin
cc SchemaInfoCommand
cc SchemaInfoEnum
cc SchemaInfoEnumMember
cc SchemaInfoEvent
cc SchemaInfoObject
cc SchemaInfoObjectMember
cc SchemaInfoObjectVariant
cc SchemaMetaType
cc SecretCommonProperties
cc SecretKeyringProperties
cc SecretProperties
cc SetPasswordAction
cc SetPasswordOptions
cc SetPasswordOptionsVnc
cc SevAttestationReport
cc SevCapability
cc SevGuestProperties
cc SevInfo
cc SevLaunchMeasureInfo
cc SevState
cc SgxEPC
cc SgxEPCDeviceInfo
cc SgxEPCDeviceInfoWrapper
cc SgxEPCProperties
cc ShutdownAction
cc ShutdownCause
cc SmbiosEntryPointType
cc SnapshotInfo
cc SocketAddress
cc SocketAddressLegacy
cc SocketAddressType
cc SpiceBasicInfo
cc SpiceChannel
cc SpiceInfo
cc SpiceQueryMouseMode
cc SpiceServerInfo
cc SshHostKeyCheck
cc SshHostKeyCheckHashType
cc SshHostKeyCheckMode
cc SshHostKeyHash
cc StatusInfo
cc StrOrNull
cc String
cc StringWrapper
cc SysEmuTarget
cc TargetInfo
cc ThrottleGroupProperties
cc ThrottleLimits
cc TlsCredsAnonProperties
cc TlsCredsProperties
cc TlsCredsPskProperties
cc TlsCredsX509Properties
cc TpmModel
cc TpmType
cc TpmTypeOptions
cc TraceEventInfo
cc TraceEventState
cc TransactionAction
cc TransactionActionKind
cc TransactionProperties
cc UnixSocketAddress
cc UnixSocketAddressWrapper
cc UuidInfo
cc VersionInfo
cc VersionTriple
cc VfioStats
cc VirtioMEMDeviceInfo
cc VirtioMEMDeviceInfoWrapper
cc VirtioPMEMDeviceInfo
cc VirtioPMEMDeviceInfoWrapper
cc VncBasicInfo
cc VncClientInfo
cc VncInfo
cc VncInfo2
cc VncPrimaryAuth
cc VncServerInfo
cc VncServerInfo2
cc VncVencryptSubAuth
cc VsockSocketAddress
cc VsockSocketAddressWrapper
cc WatchdogAction
cc YankInstance
cc YankInstanceBlockNode
cc YankInstanceChardev
cc YankInstanceType
lc a
lc aarch64
lc abort
lc aborting
lc abs
lc absolute
lc absolute-paths
lc abstract
lc accept
lc access-bandwidth
lc access-latency
lc action
lc action-required
lc actions
lc active
lc active-l1
lc active-l2
lc actual
lc actual-size
lc adapter-type
lc add-fd
lc addr
lc address
lc addresses
lc aes
lc aes-128
lc aes-192
lc aes-256
lc again
lc aio
lc aio-max-batch
lc alias
lc alias-of
lc align
lc aligned-accesses
lc all
lc allocated-clusters
lc allocation-depth
lc allow
lc allow-oob
lc allow-other
lc allow-write-only-overlay
lc alpha
lc alsa
lc alt
lc alt-alt
lc alternate
lc always
lc amend
lc amount-exceeded
lc announce-initial
lc announce-max
lc announce-rounds
lc announce-self
lc announce-step
lc api-major
lc api-minor
lc apostrophe
lc append
lc arch
lc arg-type
lc arg1
lc arg2
lc arg3
lc arg4
lc arg5
lc arm
lc array
lc associativity
lc asterisk
lc audiodev
lc audiomute
lc audionext
lc audioplay
lc audioprev
lc audiostop
lc auth
lc auth-client-required
lc authz-list
lc authz-listfile
lc authz-pam
lc authz-simple
lc auto
lc auto-converge
lc auto-dismiss
lc auto-finalize
lc auto-read-only
lc autoneg
lc avr
lc axis
lc b
lc backend
lc background
lc background-snapshot
lc backing
lc backing-file
lc backing-filename
lc backing-filename-format
lc backing-fmt
lc backing-image
lc backslash
lc backspace
lc backup
lc balloon
lc bandwidth
lc bar
lc bar0
lc bar1
lc bar2
lc bar3
lc bar4
lc bar5
lc base
lc base-memory
lc base-node
lc base64
lc before
lc begin
lc behind
lc bind
lc bins
lc bitmap
lc bitmap-directory
lc bitmap-mode
lc bitmaps
lc blk
lc blkdebug
lc blklogwrites
lc blkreplay
lc blkverify
lc block
lc block-backend
lc block-bitmap-mapping
lc block-commit
lc block-dirty-bitmap-add
lc block-dirty-bitmap-clear
lc block-dirty-bitmap-disable
lc block-dirty-bitmap-enable
lc block-dirty-bitmap-merge
lc block-dirty-bitmap-remove
lc block-driver
lc block-export-add
lc block-export-del
lc block-incremental
lc block-job
lc block-job-cancel
lc block-job-complete
lc block-job-dismiss
lc block-job-finalize
lc block-job-pause
lc block-job-resume
lc block-job-set-speed
lc block-latency-histogram-set
lc block-node
lc block-set-write-threshold
lc block-size
lc block-state-zero
lc block-status
lc block-stream
lc blockdev-add
lc blockdev-backup
lc blockdev-change-medium
lc blockdev-close-tray
lc blockdev-create
lc blockdev-del
lc blockdev-insert-medium
lc blockdev-mirror
lc blockdev-open-tray
lc blockdev-remove-medium
lc blockdev-reopen
lc blockdev-snapshot
lc blockdev-snapshot-delete-internal-sync
lc blockdev-snapshot-internal-sync
lc blockdev-snapshot-sync
lc blocked-reasons
lc bochs
lc boolean
lc bootfile
lc bottom
lc boundaries
lc boundaries-flush
lc boundaries-read
lc boundaries-write
lc bps
lc bps-read
lc bps-read-max
lc bps-read-max-length
lc bps-total
lc bps-total-max
lc bps-total-max-length
lc bps-write
lc bps-write-max
lc bps-write-max-length
lc br
lc braille
lc bridge
lc broadcast-allowed
lc btn
lc buf-size
lc buffer-count
lc buffer-length
lc build-id
lc builtin
lc bus
lc buslogic
lc busy
lc busy-rate
lc button
lc bytes
lc c
lc cache
lc cache-clean-interval
lc cache-miss
lc cache-miss-rate
lc cache-size
lc cached
lc calc-dirty-rate
lc calc-time
lc calculator
lc can-bus
lc can-host-socketcan
lc canbus
lc cancel
lc cancel-path
lc cancelled
lc cancelling
lc capabilities
lc capability
lc case
lc cast5-128
lc cbc
lc cbitpos
lc cephx
lc cert-chain
lc cert-subject
lc change-backing-file
lc change-vnc-password
lc channel-id
lc channel-type
lc channels
lc chardev
lc chardev-add
lc chardev-change
lc chardev-remove
lc chardev-send-break
lc charset
lc check-errors
lc check-stop
lc checkpoint-ready
lc checkpoint-reply
lc checkpoint-request
lc child
lc children
lc cid
lc cipher-alg
lc cipher-mode
lc class
lc client
lc client-name
lc clients
lc clipboard
lc cloop
lc closefd
lc cluster-id
lc cluster-size
lc clusters
lc cocoa
lc colo
lc colo-compare
lc cols
lc comma
lc command
lc command-line
lc commit
lc compat
lc compiled-version
lc complete
lc completed
lc completion-errors
lc completion-mode
lc complex
lc compose
lc compress
lc compress-level
lc compress-threads
lc compress-wait-thread
lc compressed
lc compressed-clusters
lc compressed-size
lc compression
lc compression-rate
lc compression-type
lc computer
lc concluded
lc conf
lc config
lc connect
lc connect-ports
lc connected
lc connection-id
lc consistent-read
lc console
lc constant
lc cont
lc control
lc cookie
lc cookie-secret
lc cookie64
lc copy
lc copy-before-write
lc copy-mode
lc copy-on-read
lc core
lc core-id
lc coreaudio
lc cores
lc corrupt
lc corruptions
lc corruptions-fixed
lc count
lc counter
lc cpu
lc cpu-index
lc cpu-max
lc cpu-state
lc cpu-throttle-increment
lc cpu-throttle-initial
lc cpu-throttle-percentage
lc cpu-throttle-tailslow
lc cpu0-id
lc cpuid-input-eax
lc cpuid-input-ecx
lc cpuid-register
lc cpus
lc crash
lc crc32c
lc crc32c-none
lc create
lc create-type
lc created
lc cris
lc cryptodev-backend
lc cryptodev-backend-builtin
lc cryptodev-vhost-user
lc ctr
lc ctrl
lc ctrl-ctrl
lc ctrl-scrolllock
lc current
lc current-progress
lc curses
lc cut
lc d
lc d0
lc d1
lc d12
lc d120
lc d144
lc d16
lc d2
lc d288
lc d3
lc d32
lc d3des
lc d4
lc d5
lc d6
lc d64
lc d7
lc d8
lc d9
lc data
lc data-file
lc data-file-raw
lc data-type
lc date-nsec
lc date-sec
lc dbus
lc dbus-vmstate
lc debug
lc decompress-threads
lc default
lc default-cpu-type
lc default-ram-id
lc default-value
lc definition
lc delay
lc delete
lc deny
lc deprecated
lc deprecated-input
lc deprecated-output
lc depth
lc des
lc desc
lc description
lc detach
lc detect-zeroes
lc dev
lc device
lc device-id
lc device-list-properties
lc devices
lc devid
lc devname
lc dh-cert-file
lc dhcpstart
lc die-id
lc dies
lc dimm
lc dir
lc direct
lc dirty-bitmap
lc dirty-bitmaps
lc dirty-flag
lc dirty-pages-rate
lc dirty-rate
lc dirty-ring
lc dirty-sync-count
lc disabled
lc disabled-wait
lc discard
lc discard-bytes-ok
lc discard-data
lc discard-nb-failed
lc discard-nb-ok
lc disconnect
lc disk
lc dismiss
lc display
lc display-reload
lc display-update
lc dist
lc dmg
lc dns
lc dnssearch
lc domainname
lc dot
lc down
lc downscript
lc downtime
lc downtime-bytes
lc downtime-limit
lc drive-backup
lc drive-mirror
lc driver
lc driver-specific
lc drop-cache
lc drv
lc dsound
lc dsp-policy
lc dst
lc dstport
lc dump
lc dump-guest-memory
lc dump-skeys
lc duplex
lc duplicate
lc dynamic
lc dynamic-auto-read-only
lc e
lc ecb
lc edges
lc egl-headless
lc eject
lc element-type
lc elf
lc emulated
lc emulator
lc enable
lc enabled
lc encoding-rate
lc encrypt
lc encrypted
lc encryption-format
lc end
lc endpoint
lc enospc
lc enum
lc equal
lc errno
lc error
lc error-desc
lc es
lc esc
lc essiv
lc eth-dst
lc eth-src
lc eth-type
lc evdev
lc event
lc events
lc exact
lc exact-name
lc exclusive
lc existing
lc expected-downtime
lc export
lc extended-l2
lc extent-size-hint
lc extents
lc external
lc extint-loop
lc extra
lc f
lc f1
lc f10
lc f11
lc f12
lc f2
lc f3
lc f32
lc f4
lc f5
lc f6
lc f7
lc f8
lc f9
lc fail
lc failed
lc failover
lc falloc
lc family
lc fat-type
lc fatal
lc fd
lc fdname
lc fds
lc fdset-id
lc features
lc fifo
lc file
lc filename
lc filter-buffer
lc filter-dump
lc filter-mirror
lc filter-node-name
lc filter-redirector
lc filter-replay
lc filter-rewriter
lc finalize
lc find
lc finish-migrate
lc first-level
lc fixed
lc fixed-iothread
lc fixed-settings
lc flags
lc flat
lc flc
lc floppy
lc flush
lc force
lc force-share
lc force-size
lc format
lc format-specific
lc formats
lc fqdn
lc fragmented-clusters
lc frequency
lc front
lc frontend-open
lc ftp
lc ftps
lc full
lc full-backing-filename
lc full-grab
lc full-screen
lc fully-allocated
lc function
lc fuse
lc g
lc getfd
lc gid-status
lc gl
lc glob
lc gluster
lc goto-tbl
lc gpa
lc grab-on-hover
lc grab-toggle
lc granularity
lc group
lc group-id
lc group-ids
lc grouped
lc growable
lc gtk
lc guest
lc guest-panic
lc guest-panicked
lc guest-reset
lc guest-shutdown
lc guestfwd
lc guid
lc h
lc half
lc handle
lc hard
lc hash
lc hash-alg
lc head
lc header-digest
lc height
lc help
lc helper
lc henkan
lc hide
lc hierarchy
lc hiragana
lc hits
lc hmat-cache
lc hmat-lb
lc hold-time
lc home
lc host
lc host-error
lc host-key-check
lc host-nodes
lc host-qmp-quit
lc host-qmp-system-reset
lc host-signal
lc host-ui
lc hostfwd
lc hostname
lc hotpluggable
lc hotpluggable-cpus
lc hotplugged
lc hppa
lc http
lc https
lc hubid
lc hubport
lc hugetlb
lc hugetlbsize
lc human-monitor-command
lc human-readable-text
lc hwversion
lc hyper-v
lc hypervisor
lc i
lc i386
lc icount
lc id
lc id-list
lc ide
lc identical
lc identity
lc if
lc ifname
lc ignore
lc ignore-unavailable
lc image
lc image-end-offset
lc image-node-name
lc immediately
lc implements
lc in
lc in-pport
lc in-use
lc inactive
lc inactive-l1
lc inactive-l2
lc inc
lc incompatible
lc inconsistent
lc incremental
lc indev
lc index
lc individual
lc inet
lc info
lc initial
lc initiator
lc initiator-name
lc inject
lc inject-error
lc inject-nmi
lc inmigrate
lc input-barrier
lc input-linux
lc input-send-event
lc insert
lc inserted
lc instances
lc int
lc interface-id
lc interfaces
lc interleave
lc internal-error
lc interval
lc io-error
lc io-status
lc iops
lc iops-read
lc iops-read-max
lc iops-read-max-length
lc iops-size
lc iops-total
lc iops-total-max
lc iops-total-max-length
lc iops-write
lc iops-write-max
lc iops-write-max-length
lc iothread
lc iotype
lc ip
lc ip-dst
lc ip-proto
lc ip-tos
lc ipv4
lc ipv6
lc ipv6-dns
lc ipv6-host
lc ipv6-prefix
lc ipv6-prefixlen
lc irq
lc is-default
lc iscsi
lc iser
lc iter-time
lc iters
lc iv
lc ivgen-alg
lc ivgen-hash-alg
lc j
lc jack
lc job-cancel
lc job-complete
lc job-dismiss
lc job-finalize
lc job-id
lc job-pause
lc job-resume
lc json-cli
lc json-cli-hotplug
lc json-type
lc k
lc katakanahiragana
lc kdump-lzo
lc kdump-snappy
lc kdump-zlib
lc keep
lc keep-alive
lc kernel
lc kernel-hashes
lc key
lc key-offset
lc key-secret
lc keyid
lc keys
lc keyslot
lc l
lc l2-cache-entry-size
lc l2-cache-size
lc l2tpv3
lc label
lc lang1
lc lang2
lc large
lc last-mode
lc late-block-activate
lc latency
lc latency-ns
lc launch-secret
lc launch-update
lc lazy-refcounts
lc lba
lc leaks
lc leaks-fixed
lc left
lc left-command-key
lc len
lc length
lc less
lc level
lc lf
lc limit
lc limits
lc line
lc link-up
lc listen
lc live
lc load
lc loaded
lc local
lc localaddr
lc location
lc locked
lc locking
lc log
lc log-append
lc log-sector-size
lc log-size
lc log-super-update-interval
lc logappend
lc logfile
lc logical-block-size
lc lsilogic
lc luks
lc luks2
lc lun
lc m
lc m68k
lc macaddr
lc mail
lc main-header
lc main-loop
lc main-mac
lc major
lc mask
lc master-key-iters
lc match
lc max
lc max-bandwidth
lc max-chunk
lc max-connections
lc max-cpu-throttle
lc max-discard
lc max-postcopy-bandwidth
lc max-size
lc max-transfer
lc max-workers
lc max-write-zero
lc maxcpus
lc maxlen
lc mbps
lc mcast
lc md5
lc measured
lc measuring
lc mediaselect
lc mem
lc mem-path
lc memaddr
lc members
lc memdev
lc memory
lc memory-backend-epc
lc memory-backend-file
lc memory-backend-memfd
lc memory-backend-ram
lc memsave
lc menu
lc merge
lc meta-meta
lc meta-type
lc metadata
lc micro
lc microblaze
lc microblazeel
lc middle
lc migrate
lc migrate-continue
lc migrate-incoming
lc migrate-pause
lc migrate-recover
lc migrate-set-capabilities
lc migrate-set-parameters
lc migrate-start-postcopy
lc migrated
lc migration
lc migration-safe
lc minor
lc minus
lc mips
lc mips64
lc mips64el
lc mipsel
lc mirror
lc mixing-engine
lc mnonce
lc mode
lc model
lc modela
lc modelb
lc mountpoint
lc mouse
lc mouse-mode
lc mptcp
lc msg
lc msmouse
lc muhenkan
lc multicast
lc multicast-overflow
lc multicast-table
lc multifd
lc multifd-bytes
lc multifd-channels
lc multifd-compression
lc multifd-zlib-level
lc multifd-zstd-level
lc mux
lc n
lc name
lc namespace
lc native
lc nbd
lc nbd-server-add
lc nbd-server-remove
lc nbd-server-start
lc nbd-server-stop
lc net
lc netdev
lc netmap
lc never
lc new-secret
lc new-vlan-id
lc nfs
lc nic
lc nios2
lc no-flush
lc nocow
lc node
lc node-id
lc node-name
lc nodeid
lc nodelay
lc nodes
lc none
lc none-crc32c
lc normal
lc normal-bytes
lc nospace
lc null
lc null-aio
lc null-co
lc num-queues
lc numa-mem-supported
lc number
lc numeric
lc nvdimm
lc nvme
lc o
lc object
lc object-add
lc object-del
lc off
lc offset
lc ok
lc old-secret
lc on
lc on-error
lc on-source-error
lc on-success
lc on-target-error
lc once
lc oob
lc opaque
lc open
lc open-timeout
lc opened
lc operating
lc operation
lc opint-loop
lc opt-discard
lc opt-write-zero
lc option
lc options
lc or1k
lc oss
lc out
lc out-pport
lc outdev
lc overflow
lc overlap-check
lc overlay
lc p
lc p2p
lc pa
lc package
lc packet-header
lc page-cache-size
lc page-sampling
lc page-size
lc pages
lc pages-per-second
lc paging
lc panic
lc parallel
lc parallels
lc parameters
lc parent
lc parent-cid
lc pass
lc pass-discard-other
lc pass-discard-request
lc pass-discard-snapshot
lc passthrough
lc password
lc password-secret
lc passwordid
lc paste
lc path
lc pause
lc pause-before-switchover
lc paused
lc payload-offset
lc pdh
lc pef-guest
lc pending
lc period-length
lc perm
lc persistent
lc pgdn
lc pgmint-loop
lc pgup
lc pincounter
lc pipe
lc plain
lc plain64
lc play
lc plugged-memory
lc pmem
lc pmemsave
lc png
lc policy
lc poll-grow
lc poll-max-ns
lc poll-shrink
lc poll-us
lc pool
lc pop-vlan
lc port
lc portal
lc ports
lc position
lc postcopy-active
lc postcopy-blocktime
lc postcopy-bytes
lc postcopy-paused
lc postcopy-ram
lc postcopy-recover
lc postcopy-requests
lc postcopy-vcpu-blocktime
lc postmigrate
lc power
lc poweroff
lc ppc
lc ppc64
lc ppm
lc pport
lc pr-manager
lc pr-manager-helper
lc pre-switchover
lc prealloc
lc prealloc-align
lc prealloc-size
lc prealloc-threads
lc preallocate
lc preallocation
lc precopy-bytes
lc preferred
lc prefetch
lc prelaunch
lc present
lc pretty
lc primary
lc print
lc priority
lc processing
lc promiscuous
lc properties
lc property
lc props
lc protocol
lc proxy-password-secret
lc proxy-username
lc psw-addr
lc psw-mask
lc pty
lc pwritev
lc q
lc qcode
lc qcow
lc qcow2
lc qdev
lc qed
lc qemu
lc qemu-vdagent
lc qom-get
lc qom-list
lc qom-list-properties
lc qom-list-types
lc qom-path
lc qom-set
lc qom-type
lc qtest
lc query-acpi-ospm-status
lc query-balloon
lc query-block
lc query-block-exports
lc query-block-jobs
lc query-blockstats
lc query-chardev
lc query-chardev-backends
lc query-colo-status
lc query-command-line-options
lc query-commands
lc query-cpu-definitions
lc query-cpu-model-baseline
lc query-cpu-model-comparison
lc query-cpu-model-expansion
lc query-cpus-fast
lc query-current-machine
lc query-dirty-rate
lc query-display-options
lc query-dump
lc query-dump-guest-memory-capability
lc query-fdsets
lc query-gic-capabilities
lc query-hotpluggable-cpus
lc query-iothreads
lc query-jobs
lc query-kvm
lc query-machines
lc query-memdev
lc query-memory-devices
lc query-memory-size-summary
lc query-mice
lc query-migrate
lc query-migrate-capabilities
lc query-migrate-parameters
lc query-name
lc query-named-block-nodes
lc query-nodes
lc query-pci
lc query-pr-managers
lc query-qmp-schema
lc query-replay
lc query-rocker
lc query-rocker-of-dpa-flows
lc query-rocker-of-dpa-groups
lc query-rocker-ports
lc query-rx-filter
lc query-sev
lc query-sev-attestation-report
lc query-sev-capabilities
lc query-sev-launch-measure
lc query-sgx
lc query-sgx-capabilities
lc query-spice
lc query-status
lc query-target
lc query-tpm
lc query-tpm-models
lc query-tpm-types
lc query-uuid
lc query-version
lc query-vm-generation-id
lc query-vnc
lc query-vnc-servers
lc query-xen-replication-status
lc query-yank
lc queue
lc queues
lc quit
lc quorum
lc r
lc ra2
lc ra2ne
lc ram
lc raw
lc rbd
lc rdma-pin-all
lc read
lc read-bandwidth
lc read-latency
lc read-only
lc read-only-mode
lc read-pattern
lc read-write
lc read-zeroes
lc readahead
lc readahead-size
lc readline
lc readonly
lc ready
lc reason
lc reboot
lc receive-update
lc rechs
lc recipient
lc reconnect
lc reconnect-delay
lc record
lc recording
lc recursive
lc reduced-phys-bits
lc refcount-bits
lc refcount-block
lc refcount-cache-size
lc refcount-table
lc reference
lc refresh
lc regions
lc reject
lc rel
lc relaunch
lc release-ram
lc remaining
lc remote
lc removable
lc remove-fd
lc rendernode
lc repeat
lc replaces
lc replay-break
lc replay-delete-break
lc replay-seek
lc replication
lc report
lc request
lc requested-size
lc require
lc required
lc reserve
lc reset
lc resize
lc responsible-properties
lc restore-vm
lc restrict
lc result
lc resume
lc ret
lc ret-type
lc retain
lc return-path
lc rev
lc rewrite-corrupted
lc right
lc ringbuf
lc ringbuf-read
lc ringbuf-write
lc ripemd160
lc riscv32
lc riscv64
lc rng-builtin
lc rng-egd
lc rng-random
lc ro
lc rounds
lc rows
lc rtc-reset-reinjection
lc rules
lc run
lc running
lc rw
lc rx
lc rxcookie
lc rxsession
lc s
lc s16
lc s32
lc s390
lc s390-pv-guest
lc s390x
lc s8
lc safe
lc sample-pages
lc sanity-check
lc sasl
lc save-vm
lc savevm-monitor-nodes
lc screendump
lc script
lc scrolllock
lc sdl
lc seal
lc second-level
lc secondary
lc secret
lc section-size
lc sections
lc sector
lc sector-num
lc sectors-count
lc semicolon
lc send-key
lc send-update
lc serial
lc serpent-128
lc serpent-192
lc serpent-256
lc server
lc server-name
lc service
lc session-file
lc set-action
lc set-eth-dst
lc set-eth-src
lc set-numa-node
lc set-speed
lc set-state
lc set-vlan-id
lc setup
lc setup-time
lc sev-device
lc sev-guest
lc sev-inject-launch-secret
lc sgx
lc sgx-epc
lc sgx1
lc sgx2
lc sh4
lc sh4eb
lc sha1
lc sha224
lc sha256
lc sha384
lc sha512
lc share
lc shared-perm
lc shift
lc shift-shift
lc show-cursor
lc shutdown
lc shutting-down
lc side
lc sig
lc signal
lc singlestep
lc size
lc skipauth
lc skipped
lc slash
lc sleep
lc slew
lc slot
lc slot-type
lc slots
lc smb
lc smbserver
lc snapshot
lc snapshot-access
lc snapshot-delete
lc snapshot-file
lc snapshot-load
lc snapshot-node-name
lc snapshot-save
lc snapshot-table
lc snapshots
lc sndbuf
lc sock
lc socket
lc socket-address
lc socket-id
lc sockets
lc source
lc sparc
lc sparc64
lc spc
lc speed
lc spice
lc spice-app
lc spiceport
lc spicevmc
lc split
lc src
lc srcport
lc ssh
lc sslverify
lc standby
lc start
lc start-server
lc start-time
lc state
lc static
lc stats
lc status
lc stdio
lc step
lc stop
lc stopped
lc str
lc stream
lc stream-name
lc string
lc stripes
lc subformat
lc subnet-prefix
lc subordinate
lc subset
lc subsystem
lc subsystem-reset
lc subsystem-vendor
lc superset
lc suspended
lc swap-opt-cmd
lc sync
lc sysrq
lc t
lc tab
lc table-size
lc tag
lc take-child-perms
lc tap
lc target
lc tbl-id
lc tcp
lc tcp-syn-count
lc telnet
lc template
lc test
lc testdev
lc tftp
lc tftp-server-name
lc third-level
lc thread-id
lc thread-pool-max
lc thread-pool-min
lc threads
lc threshold
lc throttle
lc throttle-group
lc throttle-trigger-threshold
lc tight
lc time
lc timeout
lc timer-period
lc tls
lc tls-authz
lc tls-certs
lc tls-cipher-suites
lc tls-creds
lc tls-creds-anon
lc tls-creds-psk
lc tls-creds-x509
lc tls-hostname
lc tls-none
lc tls-plain
lc tls-port
lc tls-sasl
lc tls-vnc
lc tn3270
lc to
lc toolsversion
lc top
lc top-id
lc top-node
lc total
lc total-clusters
lc total-progress
lc total-time
lc tpm-crb
lc tpm-spapr
lc tpm-tis
lc trace-event-get-state
lc trace-event-set-state
lc transaction
lc transferred
lc transform
lc transport
lc tray-open
lc tricore
lc try-mmap
lc try-poll
lc ttl-check
lc tunnel-id
lc tunnel-lport
lc twofish-128
lc twofish-192
lc twofish-256
lc tx
lc txcookie
lc txsession
lc type
lc typename
lc u
lc u16
lc u32
lc u8
lc udp
lc ultra
lc unaligned-accesses
lc unavailable
lc unavailable-features
lc undefined
lc undo
lc unicast
lc unicast-overflow
lc unicast-table
lc uninit
lc uninitialized
lc unix
lc unknown
lc unmap
lc unmapped
lc unshare-child-perms
lc unstable
lc unstable-input
lc unstable-output
lc unstarted
lc unused
lc up
lc uri
lc url
lc use-copy-range
lc user
lc username
lc utf8
lc uuid
lc v
lc v2
lc v3
lc val
lc validate-uuid
lc value
lc values
lc variants
lc vc
lc vcpu
lc vcpu-dirty-rate
lc vcpus-count
lc vde
lc vdi
lc vectors
lc vencrypt
lc vendor
lc verify-peer
lc version
lc vfio
lc vhdx
lc vhost
lc vhost-user
lc vhost-user-blk
lc vhost-vdpa
lc vhostdev
lc vhostfd
lc vhostfds
lc vhostforce
lc virtio-mem
lc virtio-pmem
lc virtual-size
lc vlan
lc vlan-id
lc vlan-table
lc vm-clock-nsec
lc vm-clock-sec
lc vm-state-size
lc vmdk
lc vmstate
lc vmstate-loaded
lc vmstate-received
lc vmstate-send
lc vmstate-size
lc vnc
lc voices
lc volume
lc volumedown
lc volumeup
lc vote-threshold
lc vpc
lc vsock
lc vvfat
lc w
lc wait
lc wait-unplug
lc waiting
lc wake
lc wakeup-suspend-support
lc watchdog
lc watchdog-set-action
lc wav
lc wctablet
lc websocket
lc wheel-down
lc wheel-left
lc wheel-right
lc wheel-up
lc width
lc win-dmp
lc window-close
lc writable
lc write
lc write-back
lc write-bandwidth
lc write-blocking
lc write-latency
lc write-threshold
lc write-through
lc write-unchanged
lc write-zeroes
lc writeback
lc writethrough
lc x
lc x-blockdev-amend
lc x-blockdev-change
lc x-blockdev-set-iothread
lc x-bps-read
lc x-bps-read-max
lc x-bps-read-max-length
lc x-bps-total
lc x-bps-total-max
lc x-bps-total-max-length
lc x-bps-write
lc x-bps-write-max
lc x-bps-write-max-length
lc x-check-cache-dropped
lc x-checkpoint-delay
lc x-colo
lc x-colo-lost-heartbeat
lc x-debug-block-dirty-bitmap-sha256
lc x-debug-query-block-graph
lc x-dirty-bitmap
lc x-exit-preconfig
lc x-ignore-shared
lc x-iops-read
lc x-iops-read-max
lc x-iops-read-max-length
lc x-iops-size
lc x-iops-total
lc x-iops-total-max
lc x-iops-total-max-length
lc x-iops-write
lc x-iops-write-max
lc x-iops-write-max-length
lc x-origin
lc x-perf
lc x-query-irq
lc x-query-jit
lc x-query-numa
lc x-query-opcount
lc x-query-profile
lc x-query-ramblock
lc x-query-rdma
lc x-query-roms
lc x-query-usb
lc x-remote-object
lc x-use-canonical-path-for-ramblock-id
lc x509-none
lc x509-plain
lc x509-sasl
lc x509-vnc
lc xbzrle
lc xbzrle-cache
lc xbzrle-cache-size
lc xen-colo-do-checkpoint
lc xen-load-devices-state
lc xen-save-devices-state
lc xen-set-global-dirty-log
lc xen-set-replication
lc xtensa
lc xtensaeb
lc xts
lc y
lc y-origin
lc yank
lc yen
lc z
lc zero
lc zero-blocks
lc zeroed-grain
lc zlib
lc zoom-to-fit
lc zstd
mc ACPIOSTInfo
mc ACPISlotType
mc COLOExitReason
mc COLOMessage
mc COLOMode
mc COLOStatus
mc DBusVMStateProperties
mc GICCapability
mc IOThreadInfo
mc JSONType
mc KVMMissingCap
mc NFSServer
mc NFSTransport
mc PCDIMMDeviceInfo
mc PCDIMMDeviceInfoWrapper
mc PCIELinkSpeed
mc PCIELinkWidth
mc PRManagerInfo
mc QAuthZListFormat
mc QAuthZListPolicy
mc QAuthZListRule
mc QCryptoBlockAmendOptions
mc QCryptoBlockAmendOptionsLUKS
mc QCryptoBlockCreateOptions
mc QCryptoBlockCreateOptionsLUKS
mc QCryptoBlockFormat
mc QCryptoBlockInfo
mc QCryptoBlockInfoBase
mc QCryptoBlockInfoLUKS
mc QCryptoBlockInfoLUKSSlot
mc QCryptoBlockLUKSKeyslotState
mc QCryptoBlockOpenOptions
mc QCryptoBlockOptionsBase
mc QCryptoBlockOptionsLUKS
mc QCryptoBlockOptionsQCow
mc QCryptoCipherAlgorithm
mc QCryptoCipherMode
mc QCryptoHashAlgorithm
mc QCryptoIVGenAlgorithm
mc QCryptoSecretFormat
mc QCryptoTLSCredsEndpoint
mc QKeyCode
mc QKeyCodeWrapper
mc QMPCapability
mc S390CrashReason
mc SGXEPCSection
mc SGXInfo
mc SMPConfiguration
mc TPMEmulatorOptions
mc TPMEmulatorOptionsWrapper
mc TPMInfo
mc TPMPassthroughOptions
mc TPMPassthroughOptionsWrapper
mc X86CPUFeatureWordInfo
mc X86CPURegister32
mc XBZRLECacheStats
mc XDbgBlockGraph
mc XDbgBlockGraphEdge
mc XDbgBlockGraphNode
mc XDbgBlockGraphNodeType
mc ac_back
mc ac_bookmarks
mc ac_forward
mc ac_home
mc ac_refresh
mc account_failed
mc account_invalid
mc add_client
mc alt_r
mc asl_compiler_id
mc asl_compiler_rev
mc avg_flush_latency_ns
mc avg_rd_latency_ns
mc avg_rd_queue_depth
mc avg_wr_latency_ns
mc avg_wr_queue_depth
mc backing_file
mc backing_file_depth
mc block_resize
mc block_set_io_throttle
mc bps_max
mc bps_max_length
mc bps_rd
mc bps_rd_max
mc bps_rd_max_length
mc bps_wr
mc bps_wr_max
mc bps_wr_max_length
mc bracket_left
mc bracket_right
mc caps_lock
mc class_info
mc client_migrate_info
mc cluster_alloc
mc cluster_alloc_bytes
mc cluster_alloc_space
mc cluster_free
mc compare_timeout
mc cor_write
mc cow_read
mc cow_write
mc ctrl_r
mc d2_5
mc detect_zeroes
mc device_add
mc device_del
mc empty_image_prepare
mc expire_password
mc expired_scan_cycle
mc failed_flush_operations
mc failed_rd_operations
mc failed_unmap_operations
mc failed_wr_operations
mc flush_latency_histogram
mc flush_operations
mc flush_to_disk
mc flush_to_os
mc flush_total_time_ns
mc grab_all
mc grave_accent
mc host_cdrom
mc host_device
mc idle_time_ns
mc interval_length
mc invalid_flush_operations
mc invalid_rd_operations
mc invalid_unmap_operations
mc invalid_wr_operations
mc io_range
mc io_uring
mc iops_max
mc iops_max_length
mc iops_rd
mc iops_rd_max
mc iops_rd_max_length
mc iops_size
mc iops_wr
mc iops_wr_max
mc iops_wr_max_length
mc irq_pin
mc known_hosts
mc kp_0
mc kp_1
mc kp_2
mc kp_3
mc kp_4
mc kp_5
mc kp_6
mc kp_7
mc kp_8
mc kp_9
mc kp_add
mc kp_comma
mc kp_decimal
mc kp_divide
mc kp_enter
mc kp_equals
mc kp_multiply
mc kp_subtract
mc l1_grow_activate_table
mc l1_grow_alloc_table
mc l1_grow_write_table
mc l1_shrink_free_l2_clusters
mc l1_shrink_write_table
mc l1_update
mc l2_alloc_cow_read
mc l2_alloc_write
mc l2_load
mc l2_update
mc l2_update_compressed
mc legacyESX
mc max_flush_latency_ns
mc max_queue_size
mc max_rd_latency_ns
mc max_wr_latency_ns
mc mem_type_64
mc memory_range
mc meta_l
mc meta_r
mc migrate_cancel
mc min_flush_latency_ns
mc min_rd_latency_ns
mc min_wr_latency_ns
mc monolithicFlat
mc monolithicSparse
mc netdev_add
mc netdev_del
mc new_state
mc notify_dev
mc num_lock
mc oem_id
mc oem_rev
mc oem_table_id
mc pci_bridge
mc prefetchable_range
mc primary_in
mc pwritev_done
mc pwritev_rmw_after_head
mc pwritev_rmw_after_tail
mc pwritev_rmw_head
mc pwritev_rmw_tail
mc pwritev_zero
mc qdev_id
mc qmp_capabilities
mc rd_bytes
mc rd_latency_histogram
mc rd_merged
mc rd_operations
mc rd_total_time_ns
mc read_aio
mc read_backing_aio
mc read_compressed
mc refblock_alloc
mc refblock_alloc_hookup
mc refblock_alloc_switch_table
mc refblock_alloc_write
mc refblock_alloc_write_blocks
mc refblock_alloc_write_table
mc refblock_load
mc refblock_update
mc refblock_update_part
mc reftable_grow
mc reftable_load
mc reftable_update
mc sasl_username
mc scroll_lock
mc secondary_in
mc secret_keyring
mc set_link
mc set_password
mc shift_r
mc streamOptimized
mc system_powerdown
mc system_reset
mc system_wakeup
mc timed_stats
mc tray_open
mc twoGbMaxExtentFlat
mc twoGbMaxExtentSparse
mc unmap_bytes
mc unmap_merged
mc unmap_operations
mc unmap_total_time_ns
mc vmstate_load
mc vmstate_save
mc vnet_hdr
mc vnet_hdr_support
mc wr_bytes
mc wr_highest_offset
mc wr_latency_histogram
mc wr_merged
mc wr_operations
mc wr_total_time_ns
mc write_aio
mc write_compressed
mc write_threshold
mc x509_dname
mc x86_64
uc ACPI_DEVICE_OST
uc BALLOON_CHANGE
uc BLOCK_EXPORT_DELETED
uc BLOCK_IMAGE_CORRUPTED
uc BLOCK_IO_ERROR
uc BLOCK_JOB_CANCELLED
uc BLOCK_JOB_COMPLETED
uc BLOCK_JOB_ERROR
uc BLOCK_JOB_PENDING
uc BLOCK_JOB_READY
uc BLOCK_WRITE_THRESHOLD
uc COLO_EXIT
uc CPU
uc DEVICE_DELETED
uc DEVICE_TRAY_MOVED
uc DEVICE_UNPLUG_GUEST_ERROR
uc DIMM
uc DUMP_COMPLETED
uc EAX
uc EBP
uc EBX
uc ECX
uc EDI
uc EDX
uc ESI
uc ESP
uc FAILOVER_NEGOTIATED
uc GUEST_CRASHLOADED
uc GUEST_PANICKED
uc JOB_STATUS_CHANGE
uc MEMORY_DEVICE_SIZE_CHANGE
uc MEMORY_FAILURE
uc MEM_UNPLUG_ERROR
uc MIGRATION
uc MIGRATION_PASS
uc NIC_RX_FILTER_CHANGED
uc POWERDOWN
uc PR_MANAGER_STATUS_CHANGED
uc QUORUM_FAILURE
uc QUORUM_REPORT_BAD
uc RDMA_GID_STATUS_CHANGED
uc RESET
uc RESUME
uc RTC_CHANGE
uc SHUTDOWN
uc SPICE_CONNECTED
uc SPICE_DISCONNECTED
uc SPICE_INITIALIZED
uc SPICE_MIGRATE_COMPLETED
uc STOP
uc SUSPEND
uc SUSPEND_DISK
uc UNPLUG_PRIMARY
uc UUID
uc VNC_CONNECTED
uc VNC_DISCONNECTED
uc VNC_INITIALIZED
uc VSERPORT_CHANGE
uc WAKEUP
uc WATCHDOG
#37 lore raw

Hi,

Quoted 15 lines · Markus Armbruster
Quoted 5 lines · Victor Toso
Quoted 1 line

That's true, but at least to me the trade-off feels reasonable.

I don't quite get the argument why it gets harder to find. We can simply provide the actual name as reference in the generated documentation, no?

Predictable name transformation can save me detours through documentation. Being able to guess the Go bindings just from the QMP Reference Manual can be nice: it lets me write Go code with just the QMP Reference manual in view. Being able to guess the name in the QAPI schema from the name in the Go bindings can also be nice: it lets me look it up in the QMP Reference manual without jumping through the bindings documentation.

Quoted 2 lines

Or do you envisage a Go bindings manual that fully replaces the QMP Reference Manual for developers writing Go?

When the language specifies a way of documenting the code and provides tools to generate documentation (e.g: html format) from the source code, it is common that IDEs would use this to provide documentation from a given library in a simple way.

Links break that. But we might not have a choice anyway due the fact that copying GPLv2+ could potentially be a problem to license the Go module as something else.

Quoted 9 lines
Quoted 6 lines

As Kevin already pointed out that he is not planning to work on the Alias (reference below), any other idea besides the Andrea's annotation suggestion? While this is not a blocker, I agree it would be nice to be consistent.

    https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg04703.html

Let's first try to get a handle on how many schema names are problematic.

Thanks for this. I'll reply again when I went through it (if no one else beats me to it).

Cheers,
Victor

Quoted 2,501 lines
$ git-diff
scripts/qapi/expr.py +1 -0
index 5a1782b57e..804b8ab455 100644
@@ -94,6 +94,7 @@ def check_name_str(name: str, info: QAPISourceInfo, source: str) -> str:
     """
     # Reserve the entire 'q_' namespace for c_name(), and for 'q_empty'
     # and 'q_obj_*' implicit type names.
+    print("###", name)
     match = valid_name.match(name)
     if not match or c_name(name, False).startswith('q_'):
         raise QAPISemError(info, "%s has an invalid name" % source)
$ cd bld
$ python3 ../scripts/qapi-gen.py -o qapi -b ../qapi/qapi-schema.json | sort -u | awk '/^### [a-z0-9-]+$/ { print "lc", $2; next } /^### [A-Z0-9_]+$/ { print "uc", $2; next } /^### ([A-Z][a-z]+)+/ { print "cc", $2; next } { print "mc", $2 }' | sort
cc Abort
cc AbortWrapper
cc AcpiTableOptions
cc ActionCompletionMode
cc AddfdInfo
cc AnnounceParameters
cc AudioFormat
cc Audiodev
cc AudiodevAlsaOptions
cc AudiodevAlsaPerDirectionOptions
cc AudiodevCoreaudioOptions
cc AudiodevCoreaudioPerDirectionOptions
cc AudiodevDriver
cc AudiodevDsoundOptions
cc AudiodevGenericOptions
cc AudiodevJackOptions
cc AudiodevJackPerDirectionOptions
cc AudiodevOssOptions
cc AudiodevOssPerDirectionOptions
cc AudiodevPaOptions
cc AudiodevPaPerDirectionOptions
cc AudiodevPerDirectionOptions
cc AudiodevSdlOptions
cc AudiodevSdlPerDirectionOptions
cc AudiodevWavOptions
cc AuthZListFileProperties
cc AuthZListProperties
cc AuthZPAMProperties
cc AuthZSimpleProperties
cc BackupCommon
cc BackupPerf
cc BalloonInfo
cc BiosAtaTranslation
cc BitmapMigrationBitmapAlias
cc BitmapMigrationBitmapAliasTransform
cc BitmapMigrationNodeAlias
cc BitmapSyncMode
cc BlkdebugEvent
cc BlkdebugIOType
cc BlkdebugInjectErrorOptions
cc BlkdebugSetStateOptions
cc BlockDeviceInfo
cc BlockDeviceIoStatus
cc BlockDeviceStats
cc BlockDeviceTimedStats
cc BlockDirtyBitmap
cc BlockDirtyBitmapAdd
cc BlockDirtyBitmapAddWrapper
cc BlockDirtyBitmapMerge
cc BlockDirtyBitmapMergeWrapper
cc BlockDirtyBitmapOrStr
cc BlockDirtyBitmapSha256
cc BlockDirtyBitmapWrapper
cc BlockDirtyInfo
cc BlockErrorAction
cc BlockExportInfo
cc BlockExportOptions
cc BlockExportOptionsFuse
cc BlockExportOptionsNbd
cc BlockExportOptionsNbdBase
cc BlockExportOptionsVhostUserBlk
cc BlockExportRemoveMode
cc BlockExportType
cc BlockIOThrottle
cc BlockInfo
cc BlockJobInfo
cc BlockLatencyHistogramInfo
cc BlockMeasureInfo
cc BlockPermission
cc BlockStats
cc BlockStatsSpecific
cc BlockStatsSpecificFile
cc BlockStatsSpecificNvme
cc BlockdevAioOptions
cc BlockdevAmendOptions
cc BlockdevAmendOptionsLUKS
cc BlockdevAmendOptionsQcow2
cc BlockdevBackup
cc BlockdevBackupWrapper
cc BlockdevCacheInfo
cc BlockdevCacheOptions
cc BlockdevChangeReadOnlyMode
cc BlockdevCreateOptions
cc BlockdevCreateOptionsFile
cc BlockdevCreateOptionsGluster
cc BlockdevCreateOptionsLUKS
cc BlockdevCreateOptionsNfs
cc BlockdevCreateOptionsParallels
cc BlockdevCreateOptionsQcow
cc BlockdevCreateOptionsQcow2
cc BlockdevCreateOptionsQed
cc BlockdevCreateOptionsRbd
cc BlockdevCreateOptionsSsh
cc BlockdevCreateOptionsVdi
cc BlockdevCreateOptionsVhdx
cc BlockdevCreateOptionsVmdk
cc BlockdevCreateOptionsVpc
cc BlockdevDetectZeroesOptions
cc BlockdevDiscardOptions
cc BlockdevDriver
cc BlockdevOnError
cc BlockdevOptions
cc BlockdevOptionsBlkdebug
cc BlockdevOptionsBlklogwrites
cc BlockdevOptionsBlkreplay
cc BlockdevOptionsBlkverify
cc BlockdevOptionsCbw
cc BlockdevOptionsCor
cc BlockdevOptionsCurlBase
cc BlockdevOptionsCurlFtp
cc BlockdevOptionsCurlFtps
cc BlockdevOptionsCurlHttp
cc BlockdevOptionsCurlHttps
cc BlockdevOptionsFile
cc BlockdevOptionsGenericCOWFormat
cc BlockdevOptionsGenericFormat
cc BlockdevOptionsGluster
cc BlockdevOptionsIscsi
cc BlockdevOptionsLUKS
cc BlockdevOptionsNVMe
cc BlockdevOptionsNbd
cc BlockdevOptionsNfs
cc BlockdevOptionsNull
cc BlockdevOptionsPreallocate
cc BlockdevOptionsQcow
cc BlockdevOptionsQcow2
cc BlockdevOptionsQuorum
cc BlockdevOptionsRaw
cc BlockdevOptionsRbd
cc BlockdevOptionsReplication
cc BlockdevOptionsSsh
cc BlockdevOptionsThrottle
cc BlockdevOptionsVVFAT
cc BlockdevQcow2Encryption
cc BlockdevQcow2EncryptionFormat
cc BlockdevQcow2Version
cc BlockdevQcowEncryption
cc BlockdevQcowEncryptionFormat
cc BlockdevRef
cc BlockdevRefOrNull
cc BlockdevSnapshot
cc BlockdevSnapshotInternal
cc BlockdevSnapshotInternalWrapper
cc BlockdevSnapshotSync
cc BlockdevSnapshotSyncWrapper
cc BlockdevSnapshotWrapper
cc BlockdevVhdxSubformat
cc BlockdevVmdkAdapterType
cc BlockdevVmdkSubformat
cc BlockdevVpcSubformat
cc CanHostSocketcanProperties
cc ChardevBackend
cc ChardevBackendInfo
cc ChardevBackendKind
cc ChardevCommon
cc ChardevCommonWrapper
cc ChardevDBus
cc ChardevDBusWrapper
cc ChardevFile
cc ChardevFileWrapper
cc ChardevHostdev
cc ChardevHostdevWrapper
cc ChardevInfo
cc ChardevMux
cc ChardevMuxWrapper
cc ChardevQemuVDAgent
cc ChardevQemuVDAgentWrapper
cc ChardevReturn
cc ChardevRingbuf
cc ChardevRingbufWrapper
cc ChardevSocket
cc ChardevSocketWrapper
cc ChardevSpiceChannel
cc ChardevSpiceChannelWrapper
cc ChardevSpicePort
cc ChardevSpicePortWrapper
cc ChardevStdio
cc ChardevStdioWrapper
cc ChardevUdp
cc ChardevUdpWrapper
cc ChardevVC
cc ChardevVCWrapper
cc ColoCompareProperties
cc CommandInfo
cc CommandLineOptionInfo
cc CommandLineParameterInfo
cc CommandLineParameterType
cc CommandNotFound
cc CompatPolicy
cc CompatPolicyInput
cc CompatPolicyOutput
cc CompressionStats
cc CpuDefinitionInfo
cc CpuInfoFast
cc CpuInfoS390
cc CpuInstanceProperties
cc CpuModelBaselineInfo
cc CpuModelCompareInfo
cc CpuModelCompareResult
cc CpuModelExpansionInfo
cc CpuModelExpansionType
cc CpuModelInfo
cc CpuS390State
cc CryptodevBackendProperties
cc CryptodevVhostUserProperties
cc CurrentMachineParams
cc DataFormat
cc DeviceNotActive
cc DeviceNotFound
cc DirtyRateInfo
cc DirtyRateMeasureMode
cc DirtyRateStatus
cc DirtyRateVcpu
cc DisplayCocoa
cc DisplayCurses
cc DisplayDBus
cc DisplayEGLHeadless
cc DisplayGLMode
cc DisplayGTK
cc DisplayOptions
cc DisplayProtocol
cc DisplayReloadOptions
cc DisplayReloadOptionsVNC
cc DisplayReloadType
cc DisplayType
cc DisplayUpdateOptions
cc DisplayUpdateOptionsVNC
cc DisplayUpdateType
cc DriveBackup
cc DriveBackupWrapper
cc DriveMirror
cc DummyForceArrays
cc DumpGuestMemoryCapability
cc DumpGuestMemoryFormat
cc DumpQueryResult
cc DumpStatus
cc EventLoopBaseProperties
cc ExpirePasswordOptions
cc ExpirePasswordOptionsVnc
cc FailoverStatus
cc FdsetFdInfo
cc FdsetInfo
cc FilterBufferProperties
cc FilterDumpProperties
cc FilterMirrorProperties
cc FilterRedirectorProperties
cc FilterRewriterProperties
cc FloppyDriveType
cc FuseExportAllowOther
cc GenericError
cc GrabToggleKeys
cc GuestPanicAction
cc GuestPanicInformation
cc GuestPanicInformationHyperV
cc GuestPanicInformationS390
cc GuestPanicInformationType
cc GuidInfo
cc HmatCacheAssociativity
cc HmatCacheWritePolicy
cc HmatLBDataType
cc HmatLBMemoryHierarchy
cc HostMemPolicy
cc HotpluggableCPU
cc HumanReadableText
cc ImageCheck
cc ImageFormat
cc ImageInfo
cc ImageInfoSpecific
cc ImageInfoSpecificKind
cc ImageInfoSpecificLUKSWrapper
cc ImageInfoSpecificQCow2
cc ImageInfoSpecificQCow2Encryption
cc ImageInfoSpecificQCow2EncryptionBase
cc ImageInfoSpecificQCow2Wrapper
cc ImageInfoSpecificRbd
cc ImageInfoSpecificRbdWrapper
cc ImageInfoSpecificVmdk
cc ImageInfoSpecificVmdkWrapper
cc InetSocketAddress
cc InetSocketAddressBase
cc InetSocketAddressWrapper
cc InputAxis
cc InputBarrierProperties
cc InputBtnEvent
cc InputBtnEventWrapper
cc InputButton
cc InputEvent
cc InputEventKind
cc InputKeyEvent
cc InputKeyEventWrapper
cc InputLinuxProperties
cc InputMoveEvent
cc InputMoveEventWrapper
cc IntWrapper
cc IoOperationType
cc IothreadProperties
cc IscsiHeaderDigest
cc IscsiTransport
cc JobInfo
cc JobStatus
cc JobType
cc JobVerb
cc KeyValue
cc KeyValueKind
cc KvmInfo
cc LostTickPolicy
cc MachineInfo
cc MainLoopProperties
cc MapEntry
cc Memdev
cc MemoryBackendEpcProperties
cc MemoryBackendFileProperties
cc MemoryBackendMemfdProperties
cc MemoryBackendProperties
cc MemoryDeviceInfo
cc MemoryDeviceInfoKind
cc MemoryFailureAction
cc MemoryFailureFlags
cc MemoryFailureRecipient
cc MemoryInfo
cc MigrateSetParameters
cc MigrationCapability
cc MigrationCapabilityStatus
cc MigrationInfo
cc MigrationParameter
cc MigrationParameters
cc MigrationStats
cc MigrationStatus
cc MirrorCopyMode
cc MirrorSyncMode
cc MonitorMode
cc MonitorOptions
cc MouseInfo
cc MultiFDCompression
cc NameInfo
cc NbdServerAddOptions
cc NbdServerOptions
cc NetClientDriver
cc NetFilterDirection
cc NetLegacyNicOptions
cc Netdev
cc NetdevBridgeOptions
cc NetdevHubPortOptions
cc NetdevL2TPv3Options
cc NetdevNetmapOptions
cc NetdevSocketOptions
cc NetdevTapOptions
cc NetdevUserOptions
cc NetdevVdeOptions
cc NetdevVhostUserOptions
cc NetdevVhostVDPAOptions
cc NetfilterInsert
cc NetfilterProperties
cc NetworkAddressFamily
cc NewImageMode
cc NumaCpuOptions
cc NumaDistOptions
cc NumaHmatCacheOptions
cc NumaHmatLBOptions
cc NumaNodeOptions
cc NumaOptions
cc NumaOptionsType
cc ObjectOptions
cc ObjectPropertyInfo
cc ObjectType
cc ObjectTypeInfo
cc OffAutoPCIBAR
cc OnOffAuto
cc OnOffSplit
cc PanicAction
cc PciBridgeInfo
cc PciBusInfo
cc PciDeviceClass
cc PciDeviceId
cc PciDeviceInfo
cc PciInfo
cc PciMemoryRange
cc PciMemoryRegion
cc PrManagerHelperProperties
cc PreallocMode
cc QapiErrorClass
cc Qcow2BitmapInfo
cc Qcow2BitmapInfoFlags
cc Qcow2CompressionType
cc Qcow2OverlapCheckFlags
cc Qcow2OverlapCheckMode
cc Qcow2OverlapChecks
cc QtestProperties
cc QuorumOpType
cc QuorumReadPattern
cc RbdAuthMode
cc RbdEncryptionCreateOptions
cc RbdEncryptionCreateOptionsLUKS
cc RbdEncryptionCreateOptionsLUKS2
cc RbdEncryptionCreateOptionsLUKSBase
cc RbdEncryptionOptions
cc RbdEncryptionOptionsLUKS
cc RbdEncryptionOptionsLUKS2
cc RbdEncryptionOptionsLUKSBase
cc RbdImageEncryptionFormat
cc RebootAction
cc RemoteObjectProperties
cc ReplayInfo
cc ReplayMode
cc ReplicationMode
cc ReplicationStatus
cc RngEgdProperties
cc RngProperties
cc RngRandomProperties
cc RockerOfDpaFlow
cc RockerOfDpaFlowAction
cc RockerOfDpaFlowKey
cc RockerOfDpaFlowMask
cc RockerOfDpaGroup
cc RockerPort
cc RockerPortAutoneg
cc RockerPortDuplex
cc RockerSwitch
cc RunState
cc RxFilterInfo
cc RxState
cc SchemaInfo
cc SchemaInfoAlternate
cc SchemaInfoAlternateMember
cc SchemaInfoArray
cc SchemaInfoBuiltin
cc SchemaInfoCommand
cc SchemaInfoEnum
cc SchemaInfoEnumMember
cc SchemaInfoEvent
cc SchemaInfoObject
cc SchemaInfoObjectMember
cc SchemaInfoObjectVariant
cc SchemaMetaType
cc SecretCommonProperties
cc SecretKeyringProperties
cc SecretProperties
cc SetPasswordAction
cc SetPasswordOptions
cc SetPasswordOptionsVnc
cc SevAttestationReport
cc SevCapability
cc SevGuestProperties
cc SevInfo
cc SevLaunchMeasureInfo
cc SevState
cc SgxEPC
cc SgxEPCDeviceInfo
cc SgxEPCDeviceInfoWrapper
cc SgxEPCProperties
cc ShutdownAction
cc ShutdownCause
cc SmbiosEntryPointType
cc SnapshotInfo
cc SocketAddress
cc SocketAddressLegacy
cc SocketAddressType
cc SpiceBasicInfo
cc SpiceChannel
cc SpiceInfo
cc SpiceQueryMouseMode
cc SpiceServerInfo
cc SshHostKeyCheck
cc SshHostKeyCheckHashType
cc SshHostKeyCheckMode
cc SshHostKeyHash
cc StatusInfo
cc StrOrNull
cc String
cc StringWrapper
cc SysEmuTarget
cc TargetInfo
cc ThrottleGroupProperties
cc ThrottleLimits
cc TlsCredsAnonProperties
cc TlsCredsProperties
cc TlsCredsPskProperties
cc TlsCredsX509Properties
cc TpmModel
cc TpmType
cc TpmTypeOptions
cc TraceEventInfo
cc TraceEventState
cc TransactionAction
cc TransactionActionKind
cc TransactionProperties
cc UnixSocketAddress
cc UnixSocketAddressWrapper
cc UuidInfo
cc VersionInfo
cc VersionTriple
cc VfioStats
cc VirtioMEMDeviceInfo
cc VirtioMEMDeviceInfoWrapper
cc VirtioPMEMDeviceInfo
cc VirtioPMEMDeviceInfoWrapper
cc VncBasicInfo
cc VncClientInfo
cc VncInfo
cc VncInfo2
cc VncPrimaryAuth
cc VncServerInfo
cc VncServerInfo2
cc VncVencryptSubAuth
cc VsockSocketAddress
cc VsockSocketAddressWrapper
cc WatchdogAction
cc YankInstance
cc YankInstanceBlockNode
cc YankInstanceChardev
cc YankInstanceType
lc a
lc aarch64
lc abort
lc aborting
lc abs
lc absolute
lc absolute-paths
lc abstract
lc accept
lc access-bandwidth
lc access-latency
lc action
lc action-required
lc actions
lc active
lc active-l1
lc active-l2
lc actual
lc actual-size
lc adapter-type
lc add-fd
lc addr
lc address
lc addresses
lc aes
lc aes-128
lc aes-192
lc aes-256
lc again
lc aio
lc aio-max-batch
lc alias
lc alias-of
lc align
lc aligned-accesses
lc all
lc allocated-clusters
lc allocation-depth
lc allow
lc allow-oob
lc allow-other
lc allow-write-only-overlay
lc alpha
lc alsa
lc alt
lc alt-alt
lc alternate
lc always
lc amend
lc amount-exceeded
lc announce-initial
lc announce-max
lc announce-rounds
lc announce-self
lc announce-step
lc api-major
lc api-minor
lc apostrophe
lc append
lc arch
lc arg-type
lc arg1
lc arg2
lc arg3
lc arg4
lc arg5
lc arm
lc array
lc associativity
lc asterisk
lc audiodev
lc audiomute
lc audionext
lc audioplay
lc audioprev
lc audiostop
lc auth
lc auth-client-required
lc authz-list
lc authz-listfile
lc authz-pam
lc authz-simple
lc auto
lc auto-converge
lc auto-dismiss
lc auto-finalize
lc auto-read-only
lc autoneg
lc avr
lc axis
lc b
lc backend
lc background
lc background-snapshot
lc backing
lc backing-file
lc backing-filename
lc backing-filename-format
lc backing-fmt
lc backing-image
lc backslash
lc backspace
lc backup
lc balloon
lc bandwidth
lc bar
lc bar0
lc bar1
lc bar2
lc bar3
lc bar4
lc bar5
lc base
lc base-memory
lc base-node
lc base64
lc before
lc begin
lc behind
lc bind
lc bins
lc bitmap
lc bitmap-directory
lc bitmap-mode
lc bitmaps
lc blk
lc blkdebug
lc blklogwrites
lc blkreplay
lc blkverify
lc block
lc block-backend
lc block-bitmap-mapping
lc block-commit
lc block-dirty-bitmap-add
lc block-dirty-bitmap-clear
lc block-dirty-bitmap-disable
lc block-dirty-bitmap-enable
lc block-dirty-bitmap-merge
lc block-dirty-bitmap-remove
lc block-driver
lc block-export-add
lc block-export-del
lc block-incremental
lc block-job
lc block-job-cancel
lc block-job-complete
lc block-job-dismiss
lc block-job-finalize
lc block-job-pause
lc block-job-resume
lc block-job-set-speed
lc block-latency-histogram-set
lc block-node
lc block-set-write-threshold
lc block-size
lc block-state-zero
lc block-status
lc block-stream
lc blockdev-add
lc blockdev-backup
lc blockdev-change-medium
lc blockdev-close-tray
lc blockdev-create
lc blockdev-del
lc blockdev-insert-medium
lc blockdev-mirror
lc blockdev-open-tray
lc blockdev-remove-medium
lc blockdev-reopen
lc blockdev-snapshot
lc blockdev-snapshot-delete-internal-sync
lc blockdev-snapshot-internal-sync
lc blockdev-snapshot-sync
lc blocked-reasons
lc bochs
lc boolean
lc bootfile
lc bottom
lc boundaries
lc boundaries-flush
lc boundaries-read
lc boundaries-write
lc bps
lc bps-read
lc bps-read-max
lc bps-read-max-length
lc bps-total
lc bps-total-max
lc bps-total-max-length
lc bps-write
lc bps-write-max
lc bps-write-max-length
lc br
lc braille
lc bridge
lc broadcast-allowed
lc btn
lc buf-size
lc buffer-count
lc buffer-length
lc build-id
lc builtin
lc bus
lc buslogic
lc busy
lc busy-rate
lc button
lc bytes
lc c
lc cache
lc cache-clean-interval
lc cache-miss
lc cache-miss-rate
lc cache-size
lc cached
lc calc-dirty-rate
lc calc-time
lc calculator
lc can-bus
lc can-host-socketcan
lc canbus
lc cancel
lc cancel-path
lc cancelled
lc cancelling
lc capabilities
lc capability
lc case
lc cast5-128
lc cbc
lc cbitpos
lc cephx
lc cert-chain
lc cert-subject
lc change-backing-file
lc change-vnc-password
lc channel-id
lc channel-type
lc channels
lc chardev
lc chardev-add
lc chardev-change
lc chardev-remove
lc chardev-send-break
lc charset
lc check-errors
lc check-stop
lc checkpoint-ready
lc checkpoint-reply
lc checkpoint-request
lc child
lc children
lc cid
lc cipher-alg
lc cipher-mode
lc class
lc client
lc client-name
lc clients
lc clipboard
lc cloop
lc closefd
lc cluster-id
lc cluster-size
lc clusters
lc cocoa
lc colo
lc colo-compare
lc cols
lc comma
lc command
lc command-line
lc commit
lc compat
lc compiled-version
lc complete
lc completed
lc completion-errors
lc completion-mode
lc complex
lc compose
lc compress
lc compress-level
lc compress-threads
lc compress-wait-thread
lc compressed
lc compressed-clusters
lc compressed-size
lc compression
lc compression-rate
lc compression-type
lc computer
lc concluded
lc conf
lc config
lc connect
lc connect-ports
lc connected
lc connection-id
lc consistent-read
lc console
lc constant
lc cont
lc control
lc cookie
lc cookie-secret
lc cookie64
lc copy
lc copy-before-write
lc copy-mode
lc copy-on-read
lc core
lc core-id
lc coreaudio
lc cores
lc corrupt
lc corruptions
lc corruptions-fixed
lc count
lc counter
lc cpu
lc cpu-index
lc cpu-max
lc cpu-state
lc cpu-throttle-increment
lc cpu-throttle-initial
lc cpu-throttle-percentage
lc cpu-throttle-tailslow
lc cpu0-id
lc cpuid-input-eax
lc cpuid-input-ecx
lc cpuid-register
lc cpus
lc crash
lc crc32c
lc crc32c-none
lc create
lc create-type
lc created
lc cris
lc cryptodev-backend
lc cryptodev-backend-builtin
lc cryptodev-vhost-user
lc ctr
lc ctrl
lc ctrl-ctrl
lc ctrl-scrolllock
lc current
lc current-progress
lc curses
lc cut
lc d
lc d0
lc d1
lc d12
lc d120
lc d144
lc d16
lc d2
lc d288
lc d3
lc d32
lc d3des
lc d4
lc d5
lc d6
lc d64
lc d7
lc d8
lc d9
lc data
lc data-file
lc data-file-raw
lc data-type
lc date-nsec
lc date-sec
lc dbus
lc dbus-vmstate
lc debug
lc decompress-threads
lc default
lc default-cpu-type
lc default-ram-id
lc default-value
lc definition
lc delay
lc delete
lc deny
lc deprecated
lc deprecated-input
lc deprecated-output
lc depth
lc des
lc desc
lc description
lc detach
lc detect-zeroes
lc dev
lc device
lc device-id
lc device-list-properties
lc devices
lc devid
lc devname
lc dh-cert-file
lc dhcpstart
lc die-id
lc dies
lc dimm
lc dir
lc direct
lc dirty-bitmap
lc dirty-bitmaps
lc dirty-flag
lc dirty-pages-rate
lc dirty-rate
lc dirty-ring
lc dirty-sync-count
lc disabled
lc disabled-wait
lc discard
lc discard-bytes-ok
lc discard-data
lc discard-nb-failed
lc discard-nb-ok
lc disconnect
lc disk
lc dismiss
lc display
lc display-reload
lc display-update
lc dist
lc dmg
lc dns
lc dnssearch
lc domainname
lc dot
lc down
lc downscript
lc downtime
lc downtime-bytes
lc downtime-limit
lc drive-backup
lc drive-mirror
lc driver
lc driver-specific
lc drop-cache
lc drv
lc dsound
lc dsp-policy
lc dst
lc dstport
lc dump
lc dump-guest-memory
lc dump-skeys
lc duplex
lc duplicate
lc dynamic
lc dynamic-auto-read-only
lc e
lc ecb
lc edges
lc egl-headless
lc eject
lc element-type
lc elf
lc emulated
lc emulator
lc enable
lc enabled
lc encoding-rate
lc encrypt
lc encrypted
lc encryption-format
lc end
lc endpoint
lc enospc
lc enum
lc equal
lc errno
lc error
lc error-desc
lc es
lc esc
lc essiv
lc eth-dst
lc eth-src
lc eth-type
lc evdev
lc event
lc events
lc exact
lc exact-name
lc exclusive
lc existing
lc expected-downtime
lc export
lc extended-l2
lc extent-size-hint
lc extents
lc external
lc extint-loop
lc extra
lc f
lc f1
lc f10
lc f11
lc f12
lc f2
lc f3
lc f32
lc f4
lc f5
lc f6
lc f7
lc f8
lc f9
lc fail
lc failed
lc failover
lc falloc
lc family
lc fat-type
lc fatal
lc fd
lc fdname
lc fds
lc fdset-id
lc features
lc fifo
lc file
lc filename
lc filter-buffer
lc filter-dump
lc filter-mirror
lc filter-node-name
lc filter-redirector
lc filter-replay
lc filter-rewriter
lc finalize
lc find
lc finish-migrate
lc first-level
lc fixed
lc fixed-iothread
lc fixed-settings
lc flags
lc flat
lc flc
lc floppy
lc flush
lc force
lc force-share
lc force-size
lc format
lc format-specific
lc formats
lc fqdn
lc fragmented-clusters
lc frequency
lc front
lc frontend-open
lc ftp
lc ftps
lc full
lc full-backing-filename
lc full-grab
lc full-screen
lc fully-allocated
lc function
lc fuse
lc g
lc getfd
lc gid-status
lc gl
lc glob
lc gluster
lc goto-tbl
lc gpa
lc grab-on-hover
lc grab-toggle
lc granularity
lc group
lc group-id
lc group-ids
lc grouped
lc growable
lc gtk
lc guest
lc guest-panic
lc guest-panicked
lc guest-reset
lc guest-shutdown
lc guestfwd
lc guid
lc h
lc half
lc handle
lc hard
lc hash
lc hash-alg
lc head
lc header-digest
lc height
lc help
lc helper
lc henkan
lc hide
lc hierarchy
lc hiragana
lc hits
lc hmat-cache
lc hmat-lb
lc hold-time
lc home
lc host
lc host-error
lc host-key-check
lc host-nodes
lc host-qmp-quit
lc host-qmp-system-reset
lc host-signal
lc host-ui
lc hostfwd
lc hostname
lc hotpluggable
lc hotpluggable-cpus
lc hotplugged
lc hppa
lc http
lc https
lc hubid
lc hubport
lc hugetlb
lc hugetlbsize
lc human-monitor-command
lc human-readable-text
lc hwversion
lc hyper-v
lc hypervisor
lc i
lc i386
lc icount
lc id
lc id-list
lc ide
lc identical
lc identity
lc if
lc ifname
lc ignore
lc ignore-unavailable
lc image
lc image-end-offset
lc image-node-name
lc immediately
lc implements
lc in
lc in-pport
lc in-use
lc inactive
lc inactive-l1
lc inactive-l2
lc inc
lc incompatible
lc inconsistent
lc incremental
lc indev
lc index
lc individual
lc inet
lc info
lc initial
lc initiator
lc initiator-name
lc inject
lc inject-error
lc inject-nmi
lc inmigrate
lc input-barrier
lc input-linux
lc input-send-event
lc insert
lc inserted
lc instances
lc int
lc interface-id
lc interfaces
lc interleave
lc internal-error
lc interval
lc io-error
lc io-status
lc iops
lc iops-read
lc iops-read-max
lc iops-read-max-length
lc iops-size
lc iops-total
lc iops-total-max
lc iops-total-max-length
lc iops-write
lc iops-write-max
lc iops-write-max-length
lc iothread
lc iotype
lc ip
lc ip-dst
lc ip-proto
lc ip-tos
lc ipv4
lc ipv6
lc ipv6-dns
lc ipv6-host
lc ipv6-prefix
lc ipv6-prefixlen
lc irq
lc is-default
lc iscsi
lc iser
lc iter-time
lc iters
lc iv
lc ivgen-alg
lc ivgen-hash-alg
lc j
lc jack
lc job-cancel
lc job-complete
lc job-dismiss
lc job-finalize
lc job-id
lc job-pause
lc job-resume
lc json-cli
lc json-cli-hotplug
lc json-type
lc k
lc katakanahiragana
lc kdump-lzo
lc kdump-snappy
lc kdump-zlib
lc keep
lc keep-alive
lc kernel
lc kernel-hashes
lc key
lc key-offset
lc key-secret
lc keyid
lc keys
lc keyslot
lc l
lc l2-cache-entry-size
lc l2-cache-size
lc l2tpv3
lc label
lc lang1
lc lang2
lc large
lc last-mode
lc late-block-activate
lc latency
lc latency-ns
lc launch-secret
lc launch-update
lc lazy-refcounts
lc lba
lc leaks
lc leaks-fixed
lc left
lc left-command-key
lc len
lc length
lc less
lc level
lc lf
lc limit
lc limits
lc line
lc link-up
lc listen
lc live
lc load
lc loaded
lc local
lc localaddr
lc location
lc locked
lc locking
lc log
lc log-append
lc log-sector-size
lc log-size
lc log-super-update-interval
lc logappend
lc logfile
lc logical-block-size
lc lsilogic
lc luks
lc luks2
lc lun
lc m
lc m68k
lc macaddr
lc mail
lc main-header
lc main-loop
lc main-mac
lc major
lc mask
lc master-key-iters
lc match
lc max
lc max-bandwidth
lc max-chunk
lc max-connections
lc max-cpu-throttle
lc max-discard
lc max-postcopy-bandwidth
lc max-size
lc max-transfer
lc max-workers
lc max-write-zero
lc maxcpus
lc maxlen
lc mbps
lc mcast
lc md5
lc measured
lc measuring
lc mediaselect
lc mem
lc mem-path
lc memaddr
lc members
lc memdev
lc memory
lc memory-backend-epc
lc memory-backend-file
lc memory-backend-memfd
lc memory-backend-ram
lc memsave
lc menu
lc merge
lc meta-meta
lc meta-type
lc metadata
lc micro
lc microblaze
lc microblazeel
lc middle
lc migrate
lc migrate-continue
lc migrate-incoming
lc migrate-pause
lc migrate-recover
lc migrate-set-capabilities
lc migrate-set-parameters
lc migrate-start-postcopy
lc migrated
lc migration
lc migration-safe
lc minor
lc minus
lc mips
lc mips64
lc mips64el
lc mipsel
lc mirror
lc mixing-engine
lc mnonce
lc mode
lc model
lc modela
lc modelb
lc mountpoint
lc mouse
lc mouse-mode
lc mptcp
lc msg
lc msmouse
lc muhenkan
lc multicast
lc multicast-overflow
lc multicast-table
lc multifd
lc multifd-bytes
lc multifd-channels
lc multifd-compression
lc multifd-zlib-level
lc multifd-zstd-level
lc mux
lc n
lc name
lc namespace
lc native
lc nbd
lc nbd-server-add
lc nbd-server-remove
lc nbd-server-start
lc nbd-server-stop
lc net
lc netdev
lc netmap
lc never
lc new-secret
lc new-vlan-id
lc nfs
lc nic
lc nios2
lc no-flush
lc nocow
lc node
lc node-id
lc node-name
lc nodeid
lc nodelay
lc nodes
lc none
lc none-crc32c
lc normal
lc normal-bytes
lc nospace
lc null
lc null-aio
lc null-co
lc num-queues
lc numa-mem-supported
lc number
lc numeric
lc nvdimm
lc nvme
lc o
lc object
lc object-add
lc object-del
lc off
lc offset
lc ok
lc old-secret
lc on
lc on-error
lc on-source-error
lc on-success
lc on-target-error
lc once
lc oob
lc opaque
lc open
lc open-timeout
lc opened
lc operating
lc operation
lc opint-loop
lc opt-discard
lc opt-write-zero
lc option
lc options
lc or1k
lc oss
lc out
lc out-pport
lc outdev
lc overflow
lc overlap-check
lc overlay
lc p
lc p2p
lc pa
lc package
lc packet-header
lc page-cache-size
lc page-sampling
lc page-size
lc pages
lc pages-per-second
lc paging
lc panic
lc parallel
lc parallels
lc parameters
lc parent
lc parent-cid
lc pass
lc pass-discard-other
lc pass-discard-request
lc pass-discard-snapshot
lc passthrough
lc password
lc password-secret
lc passwordid
lc paste
lc path
lc pause
lc pause-before-switchover
lc paused
lc payload-offset
lc pdh
lc pef-guest
lc pending
lc period-length
lc perm
lc persistent
lc pgdn
lc pgmint-loop
lc pgup
lc pincounter
lc pipe
lc plain
lc plain64
lc play
lc plugged-memory
lc pmem
lc pmemsave
lc png
lc policy
lc poll-grow
lc poll-max-ns
lc poll-shrink
lc poll-us
lc pool
lc pop-vlan
lc port
lc portal
lc ports
lc position
lc postcopy-active
lc postcopy-blocktime
lc postcopy-bytes
lc postcopy-paused
lc postcopy-ram
lc postcopy-recover
lc postcopy-requests
lc postcopy-vcpu-blocktime
lc postmigrate
lc power
lc poweroff
lc ppc
lc ppc64
lc ppm
lc pport
lc pr-manager
lc pr-manager-helper
lc pre-switchover
lc prealloc
lc prealloc-align
lc prealloc-size
lc prealloc-threads
lc preallocate
lc preallocation
lc precopy-bytes
lc preferred
lc prefetch
lc prelaunch
lc present
lc pretty
lc primary
lc print
lc priority
lc processing
lc promiscuous
lc properties
lc property
lc props
lc protocol
lc proxy-password-secret
lc proxy-username
lc psw-addr
lc psw-mask
lc pty
lc pwritev
lc q
lc qcode
lc qcow
lc qcow2
lc qdev
lc qed
lc qemu
lc qemu-vdagent
lc qom-get
lc qom-list
lc qom-list-properties
lc qom-list-types
lc qom-path
lc qom-set
lc qom-type
lc qtest
lc query-acpi-ospm-status
lc query-balloon
lc query-block
lc query-block-exports
lc query-block-jobs
lc query-blockstats
lc query-chardev
lc query-chardev-backends
lc query-colo-status
lc query-command-line-options
lc query-commands
lc query-cpu-definitions
lc query-cpu-model-baseline
lc query-cpu-model-comparison
lc query-cpu-model-expansion
lc query-cpus-fast
lc query-current-machine
lc query-dirty-rate
lc query-display-options
lc query-dump
lc query-dump-guest-memory-capability
lc query-fdsets
lc query-gic-capabilities
lc query-hotpluggable-cpus
lc query-iothreads
lc query-jobs
lc query-kvm
lc query-machines
lc query-memdev
lc query-memory-devices
lc query-memory-size-summary
lc query-mice
lc query-migrate
lc query-migrate-capabilities
lc query-migrate-parameters
lc query-name
lc query-named-block-nodes
lc query-nodes
lc query-pci
lc query-pr-managers
lc query-qmp-schema
lc query-replay
lc query-rocker
lc query-rocker-of-dpa-flows
lc query-rocker-of-dpa-groups
lc query-rocker-ports
lc query-rx-filter
lc query-sev
lc query-sev-attestation-report
lc query-sev-capabilities
lc query-sev-launch-measure
lc query-sgx
lc query-sgx-capabilities
lc query-spice
lc query-status
lc query-target
lc query-tpm
lc query-tpm-models
lc query-tpm-types
lc query-uuid
lc query-version
lc query-vm-generation-id
lc query-vnc
lc query-vnc-servers
lc query-xen-replication-status
lc query-yank
lc queue
lc queues
lc quit
lc quorum
lc r
lc ra2
lc ra2ne
lc ram
lc raw
lc rbd
lc rdma-pin-all
lc read
lc read-bandwidth
lc read-latency
lc read-only
lc read-only-mode
lc read-pattern
lc read-write
lc read-zeroes
lc readahead
lc readahead-size
lc readline
lc readonly
lc ready
lc reason
lc reboot
lc receive-update
lc rechs
lc recipient
lc reconnect
lc reconnect-delay
lc record
lc recording
lc recursive
lc reduced-phys-bits
lc refcount-bits
lc refcount-block
lc refcount-cache-size
lc refcount-table
lc reference
lc refresh
lc regions
lc reject
lc rel
lc relaunch
lc release-ram
lc remaining
lc remote
lc removable
lc remove-fd
lc rendernode
lc repeat
lc replaces
lc replay-break
lc replay-delete-break
lc replay-seek
lc replication
lc report
lc request
lc requested-size
lc require
lc required
lc reserve
lc reset
lc resize
lc responsible-properties
lc restore-vm
lc restrict
lc result
lc resume
lc ret
lc ret-type
lc retain
lc return-path
lc rev
lc rewrite-corrupted
lc right
lc ringbuf
lc ringbuf-read
lc ringbuf-write
lc ripemd160
lc riscv32
lc riscv64
lc rng-builtin
lc rng-egd
lc rng-random
lc ro
lc rounds
lc rows
lc rtc-reset-reinjection
lc rules
lc run
lc running
lc rw
lc rx
lc rxcookie
lc rxsession
lc s
lc s16
lc s32
lc s390
lc s390-pv-guest
lc s390x
lc s8
lc safe
lc sample-pages
lc sanity-check
lc sasl
lc save-vm
lc savevm-monitor-nodes
lc screendump
lc script
lc scrolllock
lc sdl
lc seal
lc second-level
lc secondary
lc secret
lc section-size
lc sections
lc sector
lc sector-num
lc sectors-count
lc semicolon
lc send-key
lc send-update
lc serial
lc serpent-128
lc serpent-192
lc serpent-256
lc server
lc server-name
lc service
lc session-file
lc set-action
lc set-eth-dst
lc set-eth-src
lc set-numa-node
lc set-speed
lc set-state
lc set-vlan-id
lc setup
lc setup-time
lc sev-device
lc sev-guest
lc sev-inject-launch-secret
lc sgx
lc sgx-epc
lc sgx1
lc sgx2
lc sh4
lc sh4eb
lc sha1
lc sha224
lc sha256
lc sha384
lc sha512
lc share
lc shared-perm
lc shift
lc shift-shift
lc show-cursor
lc shutdown
lc shutting-down
lc side
lc sig
lc signal
lc singlestep
lc size
lc skipauth
lc skipped
lc slash
lc sleep
lc slew
lc slot
lc slot-type
lc slots
lc smb
lc smbserver
lc snapshot
lc snapshot-access
lc snapshot-delete
lc snapshot-file
lc snapshot-load
lc snapshot-node-name
lc snapshot-save
lc snapshot-table
lc snapshots
lc sndbuf
lc sock
lc socket
lc socket-address
lc socket-id
lc sockets
lc source
lc sparc
lc sparc64
lc spc
lc speed
lc spice
lc spice-app
lc spiceport
lc spicevmc
lc split
lc src
lc srcport
lc ssh
lc sslverify
lc standby
lc start
lc start-server
lc start-time
lc state
lc static
lc stats
lc status
lc stdio
lc step
lc stop
lc stopped
lc str
lc stream
lc stream-name
lc string
lc stripes
lc subformat
lc subnet-prefix
lc subordinate
lc subset
lc subsystem
lc subsystem-reset
lc subsystem-vendor
lc superset
lc suspended
lc swap-opt-cmd
lc sync
lc sysrq
lc t
lc tab
lc table-size
lc tag
lc take-child-perms
lc tap
lc target
lc tbl-id
lc tcp
lc tcp-syn-count
lc telnet
lc template
lc test
lc testdev
lc tftp
lc tftp-server-name
lc third-level
lc thread-id
lc thread-pool-max
lc thread-pool-min
lc threads
lc threshold
lc throttle
lc throttle-group
lc throttle-trigger-threshold
lc tight
lc time
lc timeout
lc timer-period
lc tls
lc tls-authz
lc tls-certs
lc tls-cipher-suites
lc tls-creds
lc tls-creds-anon
lc tls-creds-psk
lc tls-creds-x509
lc tls-hostname
lc tls-none
lc tls-plain
lc tls-port
lc tls-sasl
lc tls-vnc
lc tn3270
lc to
lc toolsversion
lc top
lc top-id
lc top-node
lc total
lc total-clusters
lc total-progress
lc total-time
lc tpm-crb
lc tpm-spapr
lc tpm-tis
lc trace-event-get-state
lc trace-event-set-state
lc transaction
lc transferred
lc transform
lc transport
lc tray-open
lc tricore
lc try-mmap
lc try-poll
lc ttl-check
lc tunnel-id
lc tunnel-lport
lc twofish-128
lc twofish-192
lc twofish-256
lc tx
lc txcookie
lc txsession
lc type
lc typename
lc u
lc u16
lc u32
lc u8
lc udp
lc ultra
lc unaligned-accesses
lc unavailable
lc unavailable-features
lc undefined
lc undo
lc unicast
lc unicast-overflow
lc unicast-table
lc uninit
lc uninitialized
lc unix
lc unknown
lc unmap
lc unmapped
lc unshare-child-perms
lc unstable
lc unstable-input
lc unstable-output
lc unstarted
lc unused
lc up
lc uri
lc url
lc use-copy-range
lc user
lc username
lc utf8
lc uuid
lc v
lc v2
lc v3
lc val
lc validate-uuid
lc value
lc values
lc variants
lc vc
lc vcpu
lc vcpu-dirty-rate
lc vcpus-count
lc vde
lc vdi
lc vectors
lc vencrypt
lc vendor
lc verify-peer
lc version
lc vfio
lc vhdx
lc vhost
lc vhost-user
lc vhost-user-blk
lc vhost-vdpa
lc vhostdev
lc vhostfd
lc vhostfds
lc vhostforce
lc virtio-mem
lc virtio-pmem
lc virtual-size
lc vlan
lc vlan-id
lc vlan-table
lc vm-clock-nsec
lc vm-clock-sec
lc vm-state-size
lc vmdk
lc vmstate
lc vmstate-loaded
lc vmstate-received
lc vmstate-send
lc vmstate-size
lc vnc
lc voices
lc volume
lc volumedown
lc volumeup
lc vote-threshold
lc vpc
lc vsock
lc vvfat
lc w
lc wait
lc wait-unplug
lc waiting
lc wake
lc wakeup-suspend-support
lc watchdog
lc watchdog-set-action
lc wav
lc wctablet
lc websocket
lc wheel-down
lc wheel-left
lc wheel-right
lc wheel-up
lc width
lc win-dmp
lc window-close
lc writable
lc write
lc write-back
lc write-bandwidth
lc write-blocking
lc write-latency
lc write-threshold
lc write-through
lc write-unchanged
lc write-zeroes
lc writeback
lc writethrough
lc x
lc x-blockdev-amend
lc x-blockdev-change
lc x-blockdev-set-iothread
lc x-bps-read
lc x-bps-read-max
lc x-bps-read-max-length
lc x-bps-total
lc x-bps-total-max
lc x-bps-total-max-length
lc x-bps-write
lc x-bps-write-max
lc x-bps-write-max-length
lc x-check-cache-dropped
lc x-checkpoint-delay
lc x-colo
lc x-colo-lost-heartbeat
lc x-debug-block-dirty-bitmap-sha256
lc x-debug-query-block-graph
lc x-dirty-bitmap
lc x-exit-preconfig
lc x-ignore-shared
lc x-iops-read
lc x-iops-read-max
lc x-iops-read-max-length
lc x-iops-size
lc x-iops-total
lc x-iops-total-max
lc x-iops-total-max-length
lc x-iops-write
lc x-iops-write-max
lc x-iops-write-max-length
lc x-origin
lc x-perf
lc x-query-irq
lc x-query-jit
lc x-query-numa
lc x-query-opcount
lc x-query-profile
lc x-query-ramblock
lc x-query-rdma
lc x-query-roms
lc x-query-usb
lc x-remote-object
lc x-use-canonical-path-for-ramblock-id
lc x509-none
lc x509-plain
lc x509-sasl
lc x509-vnc
lc xbzrle
lc xbzrle-cache
lc xbzrle-cache-size
lc xen-colo-do-checkpoint
lc xen-load-devices-state
lc xen-save-devices-state
lc xen-set-global-dirty-log
lc xen-set-replication
lc xtensa
lc xtensaeb
lc xts
lc y
lc y-origin
lc yank
lc yen
lc z
lc zero
lc zero-blocks
lc zeroed-grain
lc zlib
lc zoom-to-fit
lc zstd
mc ACPIOSTInfo
mc ACPISlotType
mc COLOExitReason
mc COLOMessage
mc COLOMode
mc COLOStatus
mc DBusVMStateProperties
mc GICCapability
mc IOThreadInfo
mc JSONType
mc KVMMissingCap
mc NFSServer
mc NFSTransport
mc PCDIMMDeviceInfo
mc PCDIMMDeviceInfoWrapper
mc PCIELinkSpeed
mc PCIELinkWidth
mc PRManagerInfo
mc QAuthZListFormat
mc QAuthZListPolicy
mc QAuthZListRule
mc QCryptoBlockAmendOptions
mc QCryptoBlockAmendOptionsLUKS
mc QCryptoBlockCreateOptions
mc QCryptoBlockCreateOptionsLUKS
mc QCryptoBlockFormat
mc QCryptoBlockInfo
mc QCryptoBlockInfoBase
mc QCryptoBlockInfoLUKS
mc QCryptoBlockInfoLUKSSlot
mc QCryptoBlockLUKSKeyslotState
mc QCryptoBlockOpenOptions
mc QCryptoBlockOptionsBase
mc QCryptoBlockOptionsLUKS
mc QCryptoBlockOptionsQCow
mc QCryptoCipherAlgorithm
mc QCryptoCipherMode
mc QCryptoHashAlgorithm
mc QCryptoIVGenAlgorithm
mc QCryptoSecretFormat
mc QCryptoTLSCredsEndpoint
mc QKeyCode
mc QKeyCodeWrapper
mc QMPCapability
mc S390CrashReason
mc SGXEPCSection
mc SGXInfo
mc SMPConfiguration
mc TPMEmulatorOptions
mc TPMEmulatorOptionsWrapper
mc TPMInfo
mc TPMPassthroughOptions
mc TPMPassthroughOptionsWrapper
mc X86CPUFeatureWordInfo
mc X86CPURegister32
mc XBZRLECacheStats
mc XDbgBlockGraph
mc XDbgBlockGraphEdge
mc XDbgBlockGraphNode
mc XDbgBlockGraphNodeType
mc ac_back
mc ac_bookmarks
mc ac_forward
mc ac_home
mc ac_refresh
mc account_failed
mc account_invalid
mc add_client
mc alt_r
mc asl_compiler_id
mc asl_compiler_rev
mc avg_flush_latency_ns
mc avg_rd_latency_ns
mc avg_rd_queue_depth
mc avg_wr_latency_ns
mc avg_wr_queue_depth
mc backing_file
mc backing_file_depth
mc block_resize
mc block_set_io_throttle
mc bps_max
mc bps_max_length
mc bps_rd
mc bps_rd_max
mc bps_rd_max_length
mc bps_wr
mc bps_wr_max
mc bps_wr_max_length
mc bracket_left
mc bracket_right
mc caps_lock
mc class_info
mc client_migrate_info
mc cluster_alloc
mc cluster_alloc_bytes
mc cluster_alloc_space
mc cluster_free
mc compare_timeout
mc cor_write
mc cow_read
mc cow_write
mc ctrl_r
mc d2_5
mc detect_zeroes
mc device_add
mc device_del
mc empty_image_prepare
mc expire_password
mc expired_scan_cycle
mc failed_flush_operations
mc failed_rd_operations
mc failed_unmap_operations
mc failed_wr_operations
mc flush_latency_histogram
mc flush_operations
mc flush_to_disk
mc flush_to_os
mc flush_total_time_ns
mc grab_all
mc grave_accent
mc host_cdrom
mc host_device
mc idle_time_ns
mc interval_length
mc invalid_flush_operations
mc invalid_rd_operations
mc invalid_unmap_operations
mc invalid_wr_operations
mc io_range
mc io_uring
mc iops_max
mc iops_max_length
mc iops_rd
mc iops_rd_max
mc iops_rd_max_length
mc iops_size
mc iops_wr
mc iops_wr_max
mc iops_wr_max_length
mc irq_pin
mc known_hosts
mc kp_0
mc kp_1
mc kp_2
mc kp_3
mc kp_4
mc kp_5
mc kp_6
mc kp_7
mc kp_8
mc kp_9
mc kp_add
mc kp_comma
mc kp_decimal
mc kp_divide
mc kp_enter
mc kp_equals
mc kp_multiply
mc kp_subtract
mc l1_grow_activate_table
mc l1_grow_alloc_table
mc l1_grow_write_table
mc l1_shrink_free_l2_clusters
mc l1_shrink_write_table
mc l1_update
mc l2_alloc_cow_read
mc l2_alloc_write
mc l2_load
mc l2_update
mc l2_update_compressed
mc legacyESX
mc max_flush_latency_ns
mc max_queue_size
mc max_rd_latency_ns
mc max_wr_latency_ns
mc mem_type_64
mc memory_range
mc meta_l
mc meta_r
mc migrate_cancel
mc min_flush_latency_ns
mc min_rd_latency_ns
mc min_wr_latency_ns
mc monolithicFlat
mc monolithicSparse
mc netdev_add
mc netdev_del
mc new_state
mc notify_dev
mc num_lock
mc oem_id
mc oem_rev
mc oem_table_id
mc pci_bridge
mc prefetchable_range
mc primary_in
mc pwritev_done
mc pwritev_rmw_after_head
mc pwritev_rmw_after_tail
mc pwritev_rmw_head
mc pwritev_rmw_tail
mc pwritev_zero
mc qdev_id
mc qmp_capabilities
mc rd_bytes
mc rd_latency_histogram
mc rd_merged
mc rd_operations
mc rd_total_time_ns
mc read_aio
mc read_backing_aio
mc read_compressed
mc refblock_alloc
mc refblock_alloc_hookup
mc refblock_alloc_switch_table
mc refblock_alloc_write
mc refblock_alloc_write_blocks
mc refblock_alloc_write_table
mc refblock_load
mc refblock_update
mc refblock_update_part
mc reftable_grow
mc reftable_load
mc reftable_update
mc sasl_username
mc scroll_lock
mc secondary_in
mc secret_keyring
mc set_link
mc set_password
mc shift_r
mc streamOptimized
mc system_powerdown
mc system_reset
mc system_wakeup
mc timed_stats
mc tray_open
mc twoGbMaxExtentFlat
mc twoGbMaxExtentSparse
mc unmap_bytes
mc unmap_merged
mc unmap_operations
mc unmap_total_time_ns
mc vmstate_load
mc vmstate_save
mc vnet_hdr
mc vnet_hdr_support
mc wr_bytes
mc wr_highest_offset
mc wr_latency_histogram
mc wr_merged
mc wr_operations
mc wr_total_time_ns
mc write_aio
mc write_compressed
mc write_threshold
mc x509_dname
mc x86_64
uc ACPI_DEVICE_OST
uc BALLOON_CHANGE
uc BLOCK_EXPORT_DELETED
uc BLOCK_IMAGE_CORRUPTED
uc BLOCK_IO_ERROR
uc BLOCK_JOB_CANCELLED
uc BLOCK_JOB_COMPLETED
uc BLOCK_JOB_ERROR
uc BLOCK_JOB_PENDING
uc BLOCK_JOB_READY
uc BLOCK_WRITE_THRESHOLD
uc COLO_EXIT
uc CPU
uc DEVICE_DELETED
uc DEVICE_TRAY_MOVED
uc DEVICE_UNPLUG_GUEST_ERROR
uc DIMM
uc DUMP_COMPLETED
uc EAX
uc EBP
uc EBX
uc ECX
uc EDI
uc EDX
uc ESI
uc ESP
uc FAILOVER_NEGOTIATED
uc GUEST_CRASHLOADED
uc GUEST_PANICKED
uc JOB_STATUS_CHANGE
uc MEMORY_DEVICE_SIZE_CHANGE
uc MEMORY_FAILURE
uc MEM_UNPLUG_ERROR
uc MIGRATION
uc MIGRATION_PASS
uc NIC_RX_FILTER_CHANGED
uc POWERDOWN
uc PR_MANAGER_STATUS_CHANGED
uc QUORUM_FAILURE
uc QUORUM_REPORT_BAD
uc RDMA_GID_STATUS_CHANGED
uc RESET
uc RESUME
uc RTC_CHANGE
uc SHUTDOWN
uc SPICE_CONNECTED
uc SPICE_DISCONNECTED
uc SPICE_INITIALIZED
uc SPICE_MIGRATE_COMPLETED
uc STOP
uc SUSPEND
uc SUSPEND_DISK
uc UNPLUG_PRIMARY
uc UUID
uc VNC_CONNECTED
uc VNC_DISCONNECTED
uc VNC_INITIALIZED
uc VSERPORT_CHANGE
uc WAKEUP
uc WATCHDOG
#38 lore raw
In reply to Andrea Bolognani
Quoted 13 lines · Andrea Bolognani
Quoted 1 line

Times how many naming conventions?

Yeah, I don't think requiring all possible permutations to be spelled out in the schema is the way to go. That's exactly why my proposal was to offer a way to inject the semantic information that the parser can't figure out itself.

Once you have a way to inform the generator that "VNCProps" is made of the two words "vnc" and "props", and that "vnc" is an acronym, then it can generate an identifier appropriate for the target language without having to spell out anywhere that such an identifier would be VNCProps for Go and VncProps for Rust.

Quoted 4 lines

By the way, while looking around I realized that we also have to take into account things like D-Bus: the QAPI type ChardevDBus, for example, would probably translate verbatim to Go but have to be changed to ChardevDbus for Rust. Fun :)

The hardest one of all is probably

QAuthZListPolicy

which must be split 'QAuthZ' + 'List' + 'Policy' - the trailing uppercase ruins all heuristics you can come up with, as there's no viable way to distinguish that scenario from 'ChardevDBus' which is 'Chardev' + 'DBus' not 'ChardevD' + 'Bus'

Quoted 3 lines

Revised proposal for the annotation:

ns:word-WORD-WoRD-123Word

Ugly, but we should only need this in the fairly niche scenarios, so not too pain ful to add a handful of these:

Essentially if have the schema use CamelCase with UPPERCASE acronyms, and declare two rules:

1. Split on every boundary from lower to upper
2. Acronym detection if there's a sequence of 3 uppercase letters, then split before the last uppercase.

then we'll do the right thing with the vast majority of cases:

ChardevSocket:
Rule 1: Chardev + Socket
Rule 2: Chardev + Socket

VNCProps:
Rule 1: VNCProps
Rule 2: VNC + Props

ChardevDBus
Rule 1: Chardev + DBus
Rule 2: Chardev + DBus

and fail on

QAuthZListPolicy

Rule 1: QAuth + ZList + Policy
Rule 2: QAuth + ZList + Policy

so only the last case needs ns:QAuthZ-List-Policy annotation, whcih
doesn't feel like a big burden

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#39 lore raw
Quoted 8 lines · Daniel P. Berrangé
Quoted 3 lines · Andrea Bolognani

Revised proposal for the annotation:

ns:word-WORD-WoRD-123Word

Ugly, but we should only need this in the fairly niche scenarios, so not too pain ful to add a handful of these:

Quoted 6 lines

Essentially if have the schema use CamelCase with UPPERCASE acronyms, and declare two rules:

1. Split on every boundary from lower to upper
2. Acronym detection if there's a sequence of 3 uppercase letters, then split before the last uppercase.

That should cover most of the type names, but we're still going to need to help the parser out when it comes to detecting acronyms in other contexts, such as all instances of the word "VNC" below:

{ 'enum': 'DisplayProtocol',

    'data': [ 'vnc', ... ] }

{ 'command': 'query-vnc-servers', ... }

{ 'event': 'VNC_DISCONNECTED', ... }

Quoted 7 lines

QAuthZListPolicy

Rule 1: QAuth + ZList + Policy
Rule 2: QAuth + ZList + Policy

so only the last case needs ns:QAuthZ-List-Policy annotation, whcih
doesn't feel like a big burden

Note that in my proposal the ns: part would be used exactly for cases like this one to separate the namespace part which, as you said in your other reply, needs to be preserved when generating C code but can be safely dropped when targeting a language that has actual namespace support. So the annotation would look like

Q:AuthZ-List-Policy

in this specific case. The ns: part would be optional, as a namespace is not embedded in most of the names.

It's also interesting to see how "AuthZ" is capitalized in various Go modules that implement the concept:

https://pkg.go.dev/search?limit=50&m=symbol&q=authz

Most use "Authz" rather than "AuthZ". If we get to the point where the only bit of weirdness is how we spell this specific word, I think I'll be able to live with it :)

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
#40 lore raw
In reply to Daniel P. Berrangé
Quoted 14 lines · Daniel P. Berrangé
Quoted 13 lines · Andrea Bolognani
Quoted 1 line

Times how many naming conventions?

Yeah, I don't think requiring all possible permutations to be spelled out in the schema is the way to go. That's exactly why my proposal was to offer a way to inject the semantic information that the parser can't figure out itself.

Once you have a way to inform the generator that "VNCProps" is made of the two words "vnc" and "props", and that "vnc" is an acronym, then it can generate an identifier appropriate for the target language without having to spell out anywhere that such an identifier would be VNCProps for Go and VncProps for Rust.

Quoted 4 lines
Quoted 4 lines

By the way, while looking around I realized that we also have to take into account things like D-Bus: the QAPI type ChardevDBus, for example, would probably translate verbatim to Go but have to be changed to ChardevDbus for Rust. Fun :)

For what it's worth, I prefer Rust's style, because I hate downcasing tails of abbreviations a lot less than I hate WORDSRUNTOGETHER.

Quoted 23 lines

The hardest one of all is probably

QAuthZListPolicy

which must be split 'QAuthZ' + 'List' + 'Policy' - the trailing uppercase ruins all heuristics you can come up with, as there's no viable way to distinguish that scenario from 'ChardevDBus' which is 'Chardev' + 'DBus' not 'ChardevD' + 'Bus'

Quoted 3 lines

Revised proposal for the annotation:

ns:word-WORD-WoRD-123Word

Ugly, but we should only need this in the fairly niche scenarios, so not too pain ful to add a handful of these:

Essentially if have the schema use CamelCase with UPPERCASE acronyms, and declare two rules:

1. Split on every boundary from lower to upper
2. Acronym detection if there's a sequence of 3 uppercase letters, then split before the last uppercase.

Quoted 5 lines

then we'll do the right thing with the vast majority of cases:

ChardevSocket:
Rule 1: Chardev + Socket
Rule 2: Chardev + Socket

Rule 2 isn't used here.

Quoted 4 lines

VNCProps:
Rule 1: VNCProps
Rule 2: VNC + Props

Rule 2 is used here.

For Rust-style VncProps, rule 2 is again unused.

Quoted 2 lines

ChardevDBus

Note: "DBus" is not an (upper-case) acronym. It's a proper name grown from an abbreviation of "desktop bus".

Quoted 2 lines

Rule 1: Chardev + DBus
Rule 2: Chardev + DBus

Rule 2 isn't used here, either.

Quoted 9 lines

and fail on

QAuthZListPolicy

Rule 1: QAuth + ZList + Policy
Rule 2: QAuth + ZList + Policy

so only the last case needs ns:QAuthZ-List-Policy annotation, whcih
doesn't feel like a big burden

I think there are two interesting sub-problems within the larger problem of mapping QAPI names to "nice" target language identifiers.

1. Splitting words run together (mostly camels, but also mistakes like logappend).

2. Adjust case in words for the target language

For 1., we can rely on word separators '-', '_' and change from lower to upper case. Cases where that doesn't suffice remain.

Adopting Go-style camels for QAPI will add to them, but something like your rule 2 above should mitigate.

What about this simple greedy algorithm:

funny_words = [QAuthZ, ...]

tail = name
words = []
while tail != "":
if tail starts with any of the words in funny_words:
next_word = that member of funny_words
else:
next_word = tail up to next separator or change from lower
to upper case
append next_word to tail
tail = remainder of tail with leading separator stripped

For 2., adjusting to all lower case (like "vnc", "props", "dbus"), all upper case (like "VNC", "PROPS", "DBUS"), and capitalized (like "Vnc", "Props", "Dbus") is trivial. Adjusting to capitalized except for funny words (like "VNC", "Props", "DBus") is not. Need a list of funny words, I guess.

Perhaps a single list of funnies would do for 1. and for 2.

#41 lore raw
In reply to Andrea Bolognani

Hi!

Sorry for taking some time to reply.

Quoted 9 lines · Andrea Bolognani
Quoted 2 lines · Victor Toso

Thanks for taking a look, let me know if you have questions, ideas or suggestions.

Full disclosure: I have only given the actual implementation a very cursory look so far, and I've focused on the generated Go API instead.

Overall things look pretty good.

Glad to hear.

Quoted 10 lines

One concern that I have is about naming struct members: things like SpiceInfo.MouseMode and most others are translated from the QAPI schema exactly the way you'd expect them, but for example ChardevCommon.Logappend doesn't look quite right. Of course there's no way to programmatically figure out what to capitalize, but maybe there's room for adding this kind of information in the form of additional annotations or something like that? Same for the various structs or members that have unexpectedly-capitalized "Tls" or "Vnc" in them.

Quoted 2 lines

To be clear, I don't think the above is a blocker - just something to be aware of, and think about.

There was a good discussion around this with Markus so I don't want to break it in another thread.

I'm happy that you have found those inconsistencies. I'll reply on the other thread about it but I don't mind working towards fixing it, either at code generator level or at QAPI level.

Quoted 24 lines

My biggest concern is about the interface offered for commands.

Based on the example you have in the README and how commands are defined, invoking (a simplified version of) the trace-event-get-state command would look like

  cmd := Command{
      Name: "trace-event-get-state",
      Arg: TraceEventGetStateCommand{
          Name: "qemu_memalign",
      },
  }
  qmp_input, _ := json.Marshal(&cmd)
  // qmp_input now contains
  //   {"execute":"trace-event-get-state","arguments":{"name":"qemu_memalign"}}
  // do something with it

qmp_output :=
([]byte)({"return":{"name":"qemu_memalign","state":"disabled"}})
ret := cmd.GetReturnType()
_ = json.Unmarshal(qmp_output, &ret)
// ret is a CommandResult instance whose Value member can be cast // to a TraceEventInfo struct

Quoted 2 lines

First of all, from an application's point of view there are way too many steps involved:

It can actually get worse. I've used a lot of nested struct to define a Base type for a given Type. In Go, If you try to initialize a Type that has a nested Struct, you'll need to use the nested struct Type as field name and this is too verbose.

See https://github.com/golang/go/issues/29438 (merged with:
https://github.com/golang/go/issues/12854)

The main reason that I kept it is because it maps very well with the over-the-wire protocol.

Quoted 8 lines

performing this operation should really be as
simple as

ret, _ := qmp.TraceEventGetState("qemu_memalign") // ret is a TraceEventInfo instance

That's the end state we should be working towards.

Quoted 8 lines

Of course that assumes that the "qmp" object knows where the QMP socket is, knows how to talk the QMP protocol, transparently deals with serializing and deserializing data... Plus, in some case you might want to deal with the wire transfer yourself in an application-specific manner. So it makes sense to have the basic building blocks available and then build the more ergonomic SDK on top of that - with only the first part being in scope for this series.

Right. Indeed, I thought a bit about what I want to fit into the code generator that will reside in QEMU and what we might want to develop on top of that.

The goal for this series really is generating the data types that can be converted to/from QMP messages.

I completely agree with the message below: Type validation is important at this stage.

Quoted 7 lines

Even with that in mind, the current interface is IMO problematic because of its almost complete lack of type safety. Both Command.Arg and CommandResult.Value are of type Any and CommandBase.Name, which is used to drive the JSON unmarshal logic as well as ending up on the wire when executing a command, is just a plain string.

Quoted 7 lines

I think the low-level interface should look more like

  cmd := TraceEventGetStateCommand{
      Name: "qemu_memalign",
  }
  qmp_input, _ := json.Marshal(&cmd)
  // qmp_input looks the same as before

That isn't too hard to implement and I've started with this design at first. Each QAPI Command can implement a method Name() which returns the over-the-wire name for that Command.

I'm not yet sure if this is preferable over some other syntactic sugar function that might be generated (this series) or the next layer that will be on top of this.

But I agree with you that it should be improved before reaching actual Applications.

Quoted 10 lines

qmp_output :=
([]byte)({"return":{"name":"qemu_memalign","state":"disabled"}})
ret := TraceEventInfo{}
_ = json.Unmarshal(qmp_output, &ret)
// ret is a TraceEventInfo instance

The advantages over the current implementation is that the compiler will prevent you from doing something silly like passing the wrong set of arguments to a commmand, and that the application has to explicitly spell out what kind of object it expects to get as output.

I think that, if we know all types that we can have at QAPI spec, the process of marshalling and unmarshalling should verify it. So, even if we don't change the expected interface as suggested, that work needs to be done. For some types, I've already did it, like for Unions and Alternate types.

Example: https://gitlab.com/victortoso/qapi-go/-/blob/main/pkg/qapi/unions.go#L28

This union type can have 4 values for the Any interface type. The code generator documents it to help user's out.

  | type SocketAddressLegacy struct {
  |     // Base type for this struct
  |     SocketAddressLegacyBase
  |     // Value based on @type, possible types:
  |     // * InetSocketAddressWrapper
  |     // * UnixSocketAddressWrapper
  |     // * VsockSocketAddressWrapper
  |     // * StringWrapper
  |     Value Any
  | }

On the Marshal function, I used Sprintf as a way to fetch Value's type. There are other alternatives but to the cost of adding other deps.

  | func (s SocketAddressLegacy) MarshalJSON() ([]byte, error) {
  |     base, err := json.Marshal(s.SocketAddressLegacyBase)
  |     if err != nil {
  |         return nil, err
  |     }
  |
  |     typestr := fmt.Sprintf("%T", s.Value)
  |     typestr =
  |     typestr[strings.LastIndex(typestr, ".")+1:]

...

  |     // "The branches need not cover all possible enum values"
  |     // This means that on Marshal, we can safely ignore empty values
  |     if typestr == "<nil>" {
  |         return []byte(base), nil
  |     }

And then we have some Runtime checks to be sure to avoid the scenario mismatching Value's type.

  |     // Runtime check for supported value types
  |     if typestr != "StringWrapper" &&
  |         typestr != "InetSocketAddressWrapper" &&
  |         typestr != "UnixSocketAddressWrapper" &&
  |         typestr != "VsockSocketAddressWrapper" {
  |         return nil, errors.New(fmt.Sprintf("Type is not supported: %s", typestr))
  |    }
  |    value, err := json.Marshal(s.Value)
  |    if err != nil {
  |        return nil, err
  |    }

With Alternate type, extra care was need on Unmarshal as we don't know the underlying type without looking at the message we received. That's the only reason of StrictDecode() helper function.

I'm just pointing out with above examples that I agree with you with Type safety. It is hard to infer everything at compile-time so we need some Runtime checks. Having some nicer APIs will definitely help and improve developer experience too.

Quoted 8 lines

I'm attaching an incomplete implementation that I used for playing around. It's obviously too simplistic, but hopefully it will help illustrate my point.

Dealing with errors and commands that don't have a return value might require us to have generic CommandResult wrapper after all, but we should really try as hard as we can to stick to type safe interfaces.

Agree. Many thanks again, for the review, suggestions and discussions.

Cheers,
Victor

Quoted 66 lines
Signature · 65 lines
Andrea Bolognani / Red Hat / Virtualization

package main

import (
	"encoding/json"
	"fmt"
)

type TraceEventGetStateCommand struct {
	Name string `json:"name"`
}

func (self *TraceEventGetStateCommand) MarshalJSON() ([]byte, error) {
	type Arguments TraceEventGetStateCommand
	return json.Marshal(&struct {
		Execute   string     `json:"execute"`
		Arguments *Arguments `json:"arguments"`
	}{
		Execute:   "trace-event-get-state",
		Arguments: (*Arguments)(self),
	})
}

type TraceEventInfo struct {
	Name  string `json:"name"`
	State string `json:"state"`
}

func (self *TraceEventInfo) UnmarshalJSON(data []byte) error {
	type Return TraceEventInfo
	ret := struct {
		Return Return `json:"return"`
	}{}
	err := json.Unmarshal(data, &ret)
	if err != nil {
		return err
	}
	self.Name = ret.Return.Name
	self.State = ret.Return.State
	return nil
}

func main() {
	var err error
	var qmp_input []byte
	var qmp_output []byte

	cmd := TraceEventGetStateCommand{
		Name: "qemu_memalign",
	}
	if qmp_input, err = json.Marshal(&cmd); err != nil {
		panic(err)
	}
	fmt.Printf("   cmd: %v\n", cmd)
	fmt.Printf("-> qmp_input: %v\n", string(qmp_input))

	qmp_output = ([]byte)(`{"return":{"name":"qemu_memalign","state":"disabled"}}`)
	ret := TraceEventInfo{}
	if err = json.Unmarshal(qmp_output, &ret); err != nil {
		panic(err)
	}
	fmt.Printf("<- qmp_output: %v\n", string(qmp_output))
	fmt.Printf("   ret: %v\n", ret)
}
#42 lore raw
Quoted 31 lines · Victor Toso
Quoted 22 lines · Andrea Bolognani

Based on the example you have in the README and how commands are defined, invoking (a simplified version of) the trace-event-get-state command would look like

  cmd := Command{
      Name: "trace-event-get-state",
      Arg: TraceEventGetStateCommand{
          Name: "qemu_memalign",
      },
  }
  qmp_input, _ := json.Marshal(&cmd)
  // qmp_input now contains
  //   {"execute":"trace-event-get-state","arguments":{"name":"qemu_memalign"}}
  // do something with it

qmp_output :=
([]byte)({"return":{"name":"qemu_memalign","state":"disabled"}})
ret := cmd.GetReturnType()
_ = json.Unmarshal(qmp_output, &ret)
// ret is a CommandResult instance whose Value member can be cast // to a TraceEventInfo struct

Quoted 2 lines

First of all, from an application's point of view there are way too many steps involved:

It can actually get worse. I've used a lot of nested struct to define a Base type for a given Type. In Go, If you try to initialize a Type that has a nested Struct, you'll need to use the nested struct Type as field name and this is too verbose.

Quoted 5 lines

See https://github.com/golang/go/issues/29438 (merged with:
https://github.com/golang/go/issues/12854)

The main reason that I kept it is because it maps very well with the over-the-wire protocol.

Right, I had not realized how bad things really were :)

I've noticed the use of base types and while I didn't bring it up in my initial message because the other concerns seemed of much higher importance, I actually wondered whether we need to expose them to users of the Go SDK.

I think we should flatten things. That's what happens with the C generator already, for example in

  struct InetSocketAddress {
      /* Members inherited from InetSocketAddressBase: */
      char *host;
      char *port;
      /* Own members: */
      bool has_numeric;
      bool numeric;
      /* ... */
  };

This representation mirrors the wire protocol perfectly, so I see no reason not to adopt it. Am I missing something?

Quoted 17 lines
Quoted 8 lines

performing this operation should really be as
simple as

ret, _ := qmp.TraceEventGetState("qemu_memalign") // ret is a TraceEventInfo instance

That's the end state we should be working towards.

Quoted 8 lines

Of course that assumes that the "qmp" object knows where the QMP socket is, knows how to talk the QMP protocol, transparently deals with serializing and deserializing data... Plus, in some case you might want to deal with the wire transfer yourself in an application-specific manner. So it makes sense to have the basic building blocks available and then build the more ergonomic SDK on top of that - with only the first part being in scope for this series.

Quoted 6 lines

Right. Indeed, I thought a bit about what I want to fit into the code generator that will reside in QEMU and what we might want to develop on top of that.

The goal for this series really is generating the data types that can be converted to/from QMP messages.

That's perfectly fine, and in fact I believe that splitting the whole endeavor into three parts - QMP protocol implementation, QAPI types serialization/deserialization, and a high-level SDK that gives easy access to the previous two - is the best approach.

Quoted 75 lines
Quoted 10 lines

qmp_output :=
([]byte)({"return":{"name":"qemu_memalign","state":"disabled"}})
ret := TraceEventInfo{}
_ = json.Unmarshal(qmp_output, &ret)
// ret is a TraceEventInfo instance

The advantages over the current implementation is that the compiler will prevent you from doing something silly like passing the wrong set of arguments to a commmand, and that the application has to explicitly spell out what kind of object it expects to get as output.

I think that, if we know all types that we can have at QAPI spec, the process of marshalling and unmarshalling should verify it. So, even if we don't change the expected interface as suggested, that work needs to be done. For some types, I've already did it, like for Unions and Alternate types.

Example: https://gitlab.com/victortoso/qapi-go/-/blob/main/pkg/qapi/unions.go#L28

This union type can have 4 values for the Any interface type. The code generator documents it to help user's out.

  | type SocketAddressLegacy struct {
  |     // Base type for this struct
  |     SocketAddressLegacyBase
  |     // Value based on @type, possible types:
  |     // * InetSocketAddressWrapper
  |     // * UnixSocketAddressWrapper
  |     // * VsockSocketAddressWrapper
  |     // * StringWrapper
  |     Value Any
  | }

On the Marshal function, I used Sprintf as a way to fetch Value's type. There are other alternatives but to the cost of adding other deps.

  | func (s SocketAddressLegacy) MarshalJSON() ([]byte, error) {
  |     base, err := json.Marshal(s.SocketAddressLegacyBase)
  |     if err != nil {
  |         return nil, err
  |     }
  |
  |     typestr := fmt.Sprintf("%T", s.Value)
  |     typestr =
  |     typestr[strings.LastIndex(typestr, ".")+1:]

...

  |     // "The branches need not cover all possible enum values"
  |     // This means that on Marshal, we can safely ignore empty values
  |     if typestr == "<nil>" {
  |         return []byte(base), nil
  |     }

And then we have some Runtime checks to be sure to avoid the scenario mismatching Value's type.

  |     // Runtime check for supported value types
  |     if typestr != "StringWrapper" &&
  |         typestr != "InetSocketAddressWrapper" &&
  |         typestr != "UnixSocketAddressWrapper" &&
  |         typestr != "VsockSocketAddressWrapper" {
  |         return nil, errors.New(fmt.Sprintf("Type is not supported: %s", typestr))
  |    }
  |    value, err := json.Marshal(s.Value)
  |    if err != nil {
  |        return nil, err
  |    }

With Alternate type, extra care was need on Unmarshal as we don't know the underlying type without looking at the message we received. That's the only reason of StrictDecode() helper function.

Quoted 4 lines

I'm just pointing out with above examples that I agree with you with Type safety. It is hard to infer everything at compile-time so we need some Runtime checks. Having some nicer APIs will definitely help and improve developer experience too.

I agree that in some cases build time validation is simply not possible, but this doesn't seem to be one of those cases.

For unmarshaling? Sure, especially if we keep in mind the forward compatibility scenarios that Dan has raised elsewhere in the thread, and that to be completely honest I haven't gotten around to fully digesting yet.

But when it comes to marshaling we know ahead of time that the value is going to be one of a handful of types, and we should get the compiler to spot violations of this assumption for us. Using "Any" prevents that.

If you look at libvirt-go-xml-module, alternates are handled with things like

  type DomainInterfaceSource struct {
      Ethernet  *DomainInterfaceSourceEthernet `xml:"-"`
      Server    *DomainInterfaceSourceServer   `xml:"-"`
      // ...
  }

  type DomainInterfaceSourceEthernet struct {
      IP    []DomainInterfaceIP    `xml:"ip"`
      Route []DomainInterfaceRoute `xml:"route"`
  }

  type DomainInterfaceSourceServer struct {
      Address string                      `xml:"address,attr,omitempty"`
      Port    uint                        `xml:"port,attr,omitempty"`
      Local   *DomainInterfaceSourceLocal `xml:"local"`
  }

and (un)marshaling is done like

func (a *DomainInterfaceSource) MarshalXML(e *xml.Encoder, start

xml.StartElement) error {
      if a.User != nil {
          return nil
      } else if a.Ethernet != nil {
          if len(a.Ethernet.IP) > 0 && len(a.Ethernet.Route) > 0 {
              return e.EncodeElement(a.Ethernet, start)
          }
          return nil
      } else if // ...

}

func (a *DomainInterfaceSource) UnmarshalXML(d *xml.Decoder, start

xml.StartElement) error {
      if a.User != nil {
          return d.DecodeElement(a.User, &start)
      } else if a.Ethernet != nil {
          return d.DecodeElement(a.Ethernet, &start)
      } else if // ...

}

When using these data structures, you'll then do something like

  iface := DomainInterface{
      Source: &DomainInterfaceSource{
          Network: &DomainInterfaceNetwork{
              Network: "default",
          },
      },
  }

instead of

  iface := DomainInterface{
      Source: DomainInterfaceSource{
          Type:  "network",
          Value: DomainInterfaceNetwork{
              Network: "default",
          },
      },
  }

which is more compact and, crucially, allows the compiler to catch many typing issues ahead of time. I think this approach would work really well for QAPI too.

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
#43 lore raw
Quoted 6 lines · Andrea Bolognani
Quoted 5 lines · Victor Toso
Quoted 4 lines · Andrea Bolognani

Based on the example you have in the README and how commands are defined, invoking (a simplified version of) the trace-event-get-state command would look like

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
  cmd := Command{
      Name: "trace-event-get-state",
      Arg: TraceEventGetStateCommand{
          Name: "qemu_memalign",
      },
  }

Note there is clear redundancy here between 'Name' and the struct type. IMHO the better approach would be

       cmd := TraceEventGetStateCommand{
          Name: 'qemu_memalign'
       }

and have 'Command' simply as an interface that TraceEventGetStateCommand implements. I don't think the interface would need more than a Marshal and Demarshal method, which serialize the 'Arg' and then add the Name field explicitly.

Quoted 46 lines
Quoted 20 lines
Quoted 12 lines

qmp_input, _ := json.Marshal(&cmd)
// qmp_input now contains
// {"execute":"trace-event-get-state","arguments":{"name":"qemu_memalign"}}
// do something with it

qmp_output :=
([]byte)({"return":{"name":"qemu_memalign","state":"disabled"}})
ret := cmd.GetReturnType()
_ = json.Unmarshal(qmp_output, &ret)
// ret is a CommandResult instance whose Value member can be cast // to a TraceEventInfo struct

Quoted 2 lines

First of all, from an application's point of view there are way too many steps involved:

It can actually get worse. I've used a lot of nested struct to define a Base type for a given Type. In Go, If you try to initialize a Type that has a nested Struct, you'll need to use the nested struct Type as field name and this is too verbose.

Quoted 5 lines

See https://github.com/golang/go/issues/29438 (merged with:
https://github.com/golang/go/issues/12854)

The main reason that I kept it is because it maps very well with the over-the-wire protocol.

Right, I had not realized how bad things really were :)

I've noticed the use of base types and while I didn't bring it up in my initial message because the other concerns seemed of much higher importance, I actually wondered whether we need to expose them to users of the Go SDK.

I think we should flatten things. That's what happens with the C generator already, for example in

  struct InetSocketAddress {
      /* Members inherited from InetSocketAddressBase: */
      char *host;
      char *port;
      /* Own members: */
      bool has_numeric;
      bool numeric;
      /* ... */
  };
Quoted 2 lines

This representation mirrors the wire protocol perfectly, so I see no reason not to adopt it. Am I missing something?

The main reason not to flatten is if you have scenarios where it is useful to work against the base type directly. I'm not sure that we have such a need though.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#44 lore raw
In reply to Victor Toso
Quoted 37 lines · Victor Toso

Hi,

Happy 1st April. Not a joke :) /* ugh, took me too long to send */

This series is about adding a generator in scripts/qapi to produce Go data structures that can be used to communicate with QEMU over QMP.

* Why Go?

There are quite a few Go projects that interact with QEMU over QMP and they endup using a mix of different libraries with their own code.

** Which projects?

The ones I've found so far:

- podman machine
https://github.com/containers/podman/tree/main/pkg/machine/qemu

- kata-containers (govmm)
https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm

- lxd
https://github.com/lxc/lxd/tree/master/lxd/instance/drivers

- kubevirt (plain json strings)
https://github.com/kubevirt/kubevirt

(let me know if you know others)

* But Why?

Quoted 8 lines

I'm particularly interested in 3 out of 4 of the projects above and only Kubevirt uses libvirt to handle QEMU. That means that every QEMU releases where a QMP command, event or other data struct is added, removed or changed, those projects need to check what changed in QEMU and then address those changes in their projects, if needed.

The idea behind generating Go data structures is that we can keep a Go module which can have releases that follow QEMU releases.

We need to look at "following the QEMU releases" a bit more closely.

Merging your patches gives us the capability to generate a Go interface to HEAD's version of QMP.

The obvious way for an out-of-tree Go program to use this generated Go interface is to build with a specific version of it. It can then talk QMP to any compatible QEMU version.

Compatibility with older QEMUs is not assured: stuff added since is present on the Go QMP client end, but not on the QEMU QMP server end.

Compatibility with newer QEMUs is subject to our deprecation policy:

In general features are intended to be supported indefinitely once introduced into QEMU. In the event that a feature needs to be removed, it will be listed in this section. The feature will remain functional for the release in which it was deprecated and one further release. After these two releases, the feature is liable to be removed.

So, if you stay away from deprecated stuff, you're good for two more releases at least.

Does this work for the projects you have in mind?

Aside: graceful degradation in case of incompatibility seems desirable.

Quoted 3 lines

The project that uses this Go module, only need to bump the module version and it shall receive all the changes in their own vendored code base.

Ideally, incompatible changes that affect the Go program show up as compile errors. Do they?

Quoted 17 lines

* Status

There are a few rough edges to work on but this is usable. The major thing I forgot to add is handling Error from Commands. It'll be the first thing I'll work on next week.

If you want to start using this Today you can fetch it in at

https://gitlab.com/victortoso/qapi-go/

There are quite a few tests that I took from the examples in the qapi schema. Coverage using go's cover tool is giving 28.6% of statements

I've uploaded the a static generated godoc output of the above Go module here:

Quoted 6 lines

https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/

* License

While the generator (golang.py in this series) is GPL v2, the

I'd make it v2+, just to express my displeasure with the decision to make the initial QAPI generator v2 only for no good reason at all.

Quoted 3 lines

generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.

Meh. Can't be helped, I guess.

Quoted 11 lines

* Disclaimer to reviewers

This is my first serious python project so there'll be lots of suggetions that I'll be happy to take and learn from.

Thanks for taking a look, let me know if you have questions, ideas or suggestions.

Cheers,
Victor

#45 lore raw

Hi,

Thanks for the quick review Markus. Sorry for taking quite a bit of time to get back to you.

Quoted 45 lines · Markus Armbruster
Quoted 43 lines · Victor Toso

Hi,

Happy 1st April. Not a joke :) /* ugh, took me too long to send */

This series is about adding a generator in scripts/qapi to produce Go data structures that can be used to communicate with QEMU over QMP.

* Why Go?

There are quite a few Go projects that interact with QEMU over QMP and they endup using a mix of different libraries with their own code.

** Which projects?

The ones I've found so far:

- podman machine
https://github.com/containers/podman/tree/main/pkg/machine/qemu

- kata-containers (govmm)
https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm

- lxd
https://github.com/lxc/lxd/tree/master/lxd/instance/drivers

- kubevirt (plain json strings)
https://github.com/kubevirt/kubevirt

(let me know if you know others)

* But Why?

I'm particularly interested in 3 out of 4 of the projects above and only Kubevirt uses libvirt to handle QEMU. That means that every QEMU releases where a QMP command, event or other data struct is added, removed or changed, those projects need to check what changed in QEMU and then address those changes in their projects, if needed.

Quoted 8 lines
Quoted 2 lines

The idea behind generating Go data structures is that we can keep a Go module which can have releases that follow QEMU releases.

We need to look at "following the QEMU releases" a bit more closely.

Merging your patches gives us the capability to generate a Go interface to HEAD's version of QMP.

Right, just to put it up here, it should be expected that the qapi-go project is to have releases that match QEMU's release.

Quoted 19 lines

The obvious way for an out-of-tree Go program to use this generated Go interface is to build with a specific version of it. It can then talk QMP to any compatible QEMU version.

Compatibility with older QEMUs is not assured: stuff added since is present on the Go QMP client end, but not on the QEMU QMP server end.

Compatibility with newer QEMUs is subject to our deprecation policy:

In general features are intended to be supported indefinitely once introduced into QEMU. In the event that a feature needs to be removed, it will be listed in this section. The feature will remain functional for the release in which it was deprecated and one further release. After these two releases, the feature is liable to be removed.

Quoted 4 lines

So, if you stay away from deprecated stuff, you're good for two more releases at least.

Does this work for the projects you have in mind?

It depends on how the project will be using qapi-go, so I can't say for them all.

There are projects that will be targeting specific QEMU version (e.g: Kubevirt, Kata containers) and for those, I think they don't mind only bumping qapi-go when they plan to change the target QEMU version or perhaps to keep separated binary versions for a limited amount of time (just my guess).

Some other projects like Podman, will likely be talking with the running version of QEMU they have in that host. The possibilities are quite broad here.

Quoted 2 lines

Aside: graceful degradation in case of incompatibility seems desirable.

I agree. I haven't thought much on how to handle those scenarios yet and suggestions are more than welcomed.

I know that, those projects are already hardcoding commands and expected return types by hand. My first goal is to provide well defined types, QAPI/QMP compliant, with easy to reach documentation as provided by QAPI docs.

I expect that, step by step, we can improve things all around but I don't expect it to be done all at once.

Quoted 6 lines
Quoted 3 lines

The project that uses this Go module, only need to bump the module version and it shall receive all the changes in their own vendored code base.

Ideally, incompatible changes that affect the Go program show up as compile errors. Do they?

It depends. A field/type that they were using but is removed, for sure a compile-time error.

What about a new mandatory field? If you run the unit tests in qapi-go, there will be failures but it wouldn't be compile time error. If user doesn't define a mandatory field in a Go struct, the default value is used for the over-the-wire message.

Perhaps some tooling can be developed to help users check that something they are using has changed. I'll look into it.

Quoted 28 lines
Quoted 17 lines

* Status

There are a few rough edges to work on but this is usable. The major thing I forgot to add is handling Error from Commands. It'll be the first thing I'll work on next week.

If you want to start using this Today you can fetch it in at

https://gitlab.com/victortoso/qapi-go/

There are quite a few tests that I took from the examples in the qapi schema. Coverage using go's cover tool is giving 28.6% of statements

I've uploaded the a static generated godoc output of the above Go module here:

Quoted 6 lines

https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/

* License

While the generator (golang.py in this series) is GPL v2, the

I'd make it v2+, just to express my displeasure with the decision to make the initial QAPI generator v2 only for no good reason at all.

Quoted 5 lines
Quoted 3 lines

generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.

Meh. Can't be helped, I guess.

:)

Quoted 11 lines
Quoted 11 lines

* Disclaimer to reviewers

This is my first serious python project so there'll be lots of suggetions that I'll be happy to take and learn from.

Thanks for taking a look, let me know if you have questions, ideas or suggestions.

Cheers,
Victor

Thanks again,
Victor

#46 lore raw
In reply to Markus Armbruster
Quoted 69 lines · Markus Armbruster
Quoted 37 lines · Victor Toso

Hi,

Happy 1st April. Not a joke :) /* ugh, took me too long to send */

This series is about adding a generator in scripts/qapi to produce Go data structures that can be used to communicate with QEMU over QMP.

* Why Go?

There are quite a few Go projects that interact with QEMU over QMP and they endup using a mix of different libraries with their own code.

** Which projects?

The ones I've found so far:

- podman machine
https://github.com/containers/podman/tree/main/pkg/machine/qemu

- kata-containers (govmm)
https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm

- lxd
https://github.com/lxc/lxd/tree/master/lxd/instance/drivers

- kubevirt (plain json strings)
https://github.com/kubevirt/kubevirt

(let me know if you know others)

* But Why?

Quoted 8 lines

I'm particularly interested in 3 out of 4 of the projects above and only Kubevirt uses libvirt to handle QEMU. That means that every QEMU releases where a QMP command, event or other data struct is added, removed or changed, those projects need to check what changed in QEMU and then address those changes in their projects, if needed.

The idea behind generating Go data structures is that we can keep a Go module which can have releases that follow QEMU releases.

We need to look at "following the QEMU releases" a bit more closely.

Merging your patches gives us the capability to generate a Go interface to HEAD's version of QMP.

The obvious way for an out-of-tree Go program to use this generated Go interface is to build with a specific version of it. It can then talk QMP to any compatible QEMU version.

Compatibility with older QEMUs is not assured: stuff added since is present on the Go QMP client end, but not on the QEMU QMP server end.

Compatibility with newer QEMUs is subject to our deprecation policy:

In general features are intended to be supported indefinitely once introduced into QEMU. In the event that a feature needs to be removed, it will be listed in this section. The feature will remain functional for the release in which it was deprecated and one further release. After these two releases, the feature is liable to be removed.

Quoted 4 lines

So, if you stay away from deprecated stuff, you're good for two more releases at least.

Does this work for the projects you have in mind?

It might work for some projects, but in the general case I find it pretty unappealing as a restriction. Mixing and matching new QEMU with old libvirt, or vica-verca has been an incredibly common thing todo when both developing and perhaps more importantly debugging problems. For example I have one libvirt build and I use it against any QEMU from Fedora / any RHEL-8.x update, which spans a great many QEMU releases.

I like the idea of auto-generating clients from the QAPI schema, and would like it if we were able to use this kind of approach on the libvirt side, but for that we need to be more flexible in version matching.

Our current approach to deprecation features and subsequently removing them from the QAPI schema works fine when the QAPI schema is only used internally by QEMU, not when we we expand usage of QAPI to external applications.

I think we need to figure out a way to make the QAPI schema itself be append only, while still allowing QEMU to deprecation & remove features.

For a minimum viable use case, this doesn't feel all that difficult, as conceptually instead of deleting the field from QAPI, we just need to annotate it to say when it was deleted from the QEMU side. The QAPI generator for internal QEMU usage, can omit any fields annotated as deleted in QAPI schema. The QAPI generator for external app usage, can (optionally) be told to include deleted fields ranging back to a given version number. So apps can chooses what degree of compat they wish to retain.

Apps that wish to have version compat, would of course need to write their code to be aware of which fields they need to seend for which QEMU version.

Quoted 20 lines
Quoted 20 lines

* Status

There are a few rough edges to work on but this is usable. The major thing I forgot to add is handling Error from Commands. It'll be the first thing I'll work on next week.

If you want to start using this Today you can fetch it in at

https://gitlab.com/victortoso/qapi-go/

There are quite a few tests that I took from the examples in the qapi schema. Coverage using go's cover tool is giving 28.6% of statements

I've uploaded the a static generated godoc output of the above Go module here:

https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/

Quoted 6 lines
Quoted 3 lines

* License

While the generator (golang.py in this series) is GPL v2, the

I'd make it v2+, just to express my displeasure with the decision to make the initial QAPI generator v2 only for no good reason at all.

Our policy is that all new code should be v2+ anyway, unless it was clearly derived from some pre-existing v2-only code. Upto Victor to say whether the golang.py is considered clean, or was copy+paste in any parts from existin v2-only code

Quoted 5 lines
Quoted 3 lines

generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.

Meh. Can't be helped, I guess.

This does make me wonder though whether the license of the QAPI input files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is the Go code to be considered a derived work of the QAPI JSON files. I'm not finding a clearly articulated POV on this question so far.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#47 lore raw

Hi,

Quoted 11 lines · Daniel P. Berrangé
Quoted 6 lines
Quoted 3 lines

* License

While the generator (golang.py in this series) is GPL v2, the

I'd make it v2+, just to express my displeasure with the decision to make the initial QAPI generator v2 only for no good reason at all.

Our policy is that all new code should be v2+ anyway, unless it was clearly derived from some pre-existing v2-only code. Upto Victor to say whether the golang.py is considered clean, or was copy+paste in any parts from existin v2-only code

Should be fine to consider it v2+, the generator.

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.

Meh. Can't be helped, I guess.

This does make me wonder though whether the license of the QAPI input files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is the Go code to be considered a derived work of the QAPI JSON files.

GPL does not enforce that the generated code to be GPL [0] unless the generator copies GPL code to the output. My only concern has been the fact that I am copying the documentation of QAPI specification to the Go package in order to have data structures, commands, etc. with the information provided by the specification.

[0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput

Quoted 2 lines

I'm not finding a clearly articulated POV on this question so far.

I don't find it trivial either but I've accepted that the Go data structures are fine based on [0] and the documentation can be split from the Go module (sadly!) if someone finds it to be a legal issue.

Cheers,
Victor

#48 lore raw
Quoted 37 lines · Victor Toso

Hi,

Quoted 11 lines · Daniel P. Berrangé
Quoted 6 lines
Quoted 3 lines

* License

While the generator (golang.py in this series) is GPL v2, the

I'd make it v2+, just to express my displeasure with the decision to make the initial QAPI generator v2 only for no good reason at all.

Our policy is that all new code should be v2+ anyway, unless it was clearly derived from some pre-existing v2-only code. Upto Victor to say whether the golang.py is considered clean, or was copy+paste in any parts from existin v2-only code

Should be fine to consider it v2+, the generator.

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.

Meh. Can't be helped, I guess.

This does make me wonder though whether the license of the QAPI input files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is the Go code to be considered a derived work of the QAPI JSON files.

GPL does not enforce that the generated code to be GPL [0] unless the generator copies GPL code to the output. My only concern has been the fact that I am copying the documentation of QAPI specification to the Go package in order to have data structures, commands, etc. with the information provided by the specification.

[0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput

Quoted 7 lines
Quoted 2 lines

I'm not finding a clearly articulated POV on this question so far.

I don't find it trivial either but I've accepted that the Go data structures are fine based on [0] and the documentation can be split from the Go module (sadly!) if someone finds it to be a legal issue.

Ah well that link above is actually reasonably clear:

"More generally, when a program translates its input into
some other form, the copyright status of the output inherits that of the input it was generated from. "

In our case the input is the QAPI JSON, and we're translating that into Golang. That could be read as meaning our Golang code has to be GPLv2+ licensed just as with the QAPI, not merely the docs.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#49 lore raw
Quoted 48 lines · Daniel P. Berrangé
Quoted 37 lines · Victor Toso

Hi,

Quoted 11 lines · Daniel P. Berrangé
Quoted 6 lines
Quoted 3 lines

* License

While the generator (golang.py in this series) is GPL v2, the

I'd make it v2+, just to express my displeasure with the decision to make the initial QAPI generator v2 only for no good reason at all.

Our policy is that all new code should be v2+ anyway, unless it was clearly derived from some pre-existing v2-only code. Upto Victor to say whether the golang.py is considered clean, or was copy+paste in any parts from existin v2-only code

Should be fine to consider it v2+, the generator.

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.

Meh. Can't be helped, I guess.

This does make me wonder though whether the license of the QAPI input files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is the Go code to be considered a derived work of the QAPI JSON files.

GPL does not enforce that the generated code to be GPL [0] unless the generator copies GPL code to the output. My only concern has been the fact that I am copying the documentation of QAPI specification to the Go package in order to have data structures, commands, etc. with the information provided by the specification.

[0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput

Quoted 7 lines
Quoted 2 lines

I'm not finding a clearly articulated POV on this question so far.

I don't find it trivial either but I've accepted that the Go data structures are fine based on [0] and the documentation can be split from the Go module (sadly!) if someone finds it to be a legal issue.

Ah well that link above is actually reasonably clear:

Quoted 7 lines

"More generally, when a program translates its input into
some other form, the copyright status of the output inherits that of the input it was generated from. "

In our case the input is the QAPI JSON, and we're translating that into Golang. That could be read as meaning our Golang code has to be GPLv2+ licensed just as with the QAPI, not merely the docs.

Oh but I'm forgetting that QAPI JSON can be said to be our public API interface, and so while the docs text would be GPLv2+, we can claim fair use for the interface definition in the generator and pick an arbitrary output license.

We could likely deal with the docs problem by not copying the docs directly into the generated code, instead link to the published docs on qemu.org. This would require us to improve our current docs generated anchor generation. ie currently docs link for say the struct 'NumaNodeOptions' potentially changes every time we generate it

https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#qapidoc-2416

we would need to that be something stable, ie

https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions

Then the generated Go can just do

// See QAPI docs at https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions

thus avoiding any copyright complication

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#50 lore raw

Hi,

Quoted 36 lines · Daniel P. Berrangé
Quoted 35 lines · Daniel P. Berrangé
Quoted 20 lines · Victor Toso
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.

Meh. Can't be helped, I guess.

This does make me wonder though whether the license of the QAPI input files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is the Go code to be considered a derived work of the QAPI JSON files.

GPL does not enforce that the generated code to be GPL [0] unless the generator copies GPL code to the output. My only concern has been the fact that I am copying the documentation of QAPI specification to the Go package in order to have data structures, commands, etc. with the information provided by the specification.

[0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput

Quoted 7 lines
Quoted 2 lines

I'm not finding a clearly articulated POV on this question so far.

I don't find it trivial either but I've accepted that the Go data structures are fine based on [0] and the documentation can be split from the Go module (sadly!) if someone finds it to be a legal issue.

Ah well that link above is actually reasonably clear:

"More generally, when a program translates its input into
some other form, the copyright status of the output inherits that of the input it was generated from. "

Quoted 8 lines
Quoted 3 lines

In our case the input is the QAPI JSON, and we're translating that into Golang. That could be read as meaning our Golang code has to be GPLv2+ licensed just as with the QAPI, not merely the docs.

Oh but I'm forgetting that QAPI JSON can be said to be our public API interface, and so while the docs text would be GPLv2+, we can claim fair use for the interface definition in the generator and pick an arbitrary output license.

Still, it explicit says in the section "In what cases is the output of a GPL program covered by the GPL too?" is " Only when the program copies part of itself into the output".

https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#WhatCaseIsOutputGPL

So, to my understating, even if we are consuming a GPLv2+ spec with a GPLv2+ generator, the output does not need to be GPLv2+ too, unless we are *copying* parts of the input/generator into the output - which is the case for the documentation only.

I'll raise this again with the my company's legal team to be sure.

Quoted 11 lines

We could likely deal with the docs problem by not copying the docs directly into the generated code, instead link to the published docs on qemu.org. This would require us to improve our current docs generated anchor generation. ie currently docs link for say the struct 'NumaNodeOptions' potentially changes every time we generate it

https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#qapidoc-2416

we would need to that be something stable, ie

Quoted 7 lines

https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions

Then the generated Go can just do

   // See QAPI docs at https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions

thus avoiding any copyright complication

Yes, but it would be quite sad solution. Documentation in Go is bounded to the module source code and we would be making people jump thorough links here.

I mean, if that's what we need to do, okay.

I'll keep thinking about alternatives.

Cheers,
Victor

#51 lore raw
Quoted 79 lines · Victor Toso

Hi,

Quoted 36 lines · Daniel P. Berrangé
Quoted 35 lines · Daniel P. Berrangé
Quoted 20 lines · Victor Toso
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.

Meh. Can't be helped, I guess.

This does make me wonder though whether the license of the QAPI input files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is the Go code to be considered a derived work of the QAPI JSON files.

GPL does not enforce that the generated code to be GPL [0] unless the generator copies GPL code to the output. My only concern has been the fact that I am copying the documentation of QAPI specification to the Go package in order to have data structures, commands, etc. with the information provided by the specification.

[0] https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#GPLOutput

Quoted 7 lines
Quoted 2 lines

I'm not finding a clearly articulated POV on this question so far.

I don't find it trivial either but I've accepted that the Go data structures are fine based on [0] and the documentation can be split from the Go module (sadly!) if someone finds it to be a legal issue.

Ah well that link above is actually reasonably clear:

"More generally, when a program translates its input into
some other form, the copyright status of the output inherits that of the input it was generated from. "

Quoted 8 lines
Quoted 3 lines

In our case the input is the QAPI JSON, and we're translating that into Golang. That could be read as meaning our Golang code has to be GPLv2+ licensed just as with the QAPI, not merely the docs.

Oh but I'm forgetting that QAPI JSON can be said to be our public API interface, and so while the docs text would be GPLv2+, we can claim fair use for the interface definition in the generator and pick an arbitrary output license.

Still, it explicit says in the section "In what cases is the output of a GPL program covered by the GPL too?" is " Only when the program copies part of itself into the output".

https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.html#WhatCaseIsOutputGPL

So, to my understating, even if we are consuming a GPLv2+ spec with a GPLv2+ generator, the output does not need to be GPLv2+ too, unless we are *copying* parts of the input/generator into the output - which is the case for the documentation only.

I'll raise this again with the my company's legal team to be sure.

Quoted 17 lines

We could likely deal with the docs problem by not copying the docs directly into the generated code, instead link to the published docs on qemu.org. This would require us to improve our current docs generated anchor generation. ie currently docs link for say the struct 'NumaNodeOptions' potentially changes every time we generate it

https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#qapidoc-2416

we would need to that be something stable, ie

https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions

Then the generated Go can just do

   // See QAPI docs at https://www.qemu.org/docs/master/interop/qemu-qmp-ref.html#struct-NumaNodeOptions
Quoted 7 lines
Quoted 1 line

thus avoiding any copyright complication

Yes, but it would be quite sad solution. Documentation in Go is bounded to the module source code and we would be making people jump thorough links here.

I mean, if that's what we need to do, okay.

It isn't the end of the world IMHO, as people are typically browsing docs from the online docs site, so full details are only a click away.

This is what I did in libvirt Go APIs for example

https://pkg.go.dev/libvirt.org/go/libvirt#Domain.SnapshotLookupByName

I feel the biggest impact for developers is actually the quality of the docs that exist. Time invested in better QAPI docs will have more impact on developers, than trying to eliminate the need to follow one extra hyperlink.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#52 lore raw
In reply to Daniel P. Berrangé
Quoted 73 lines · Daniel P. Berrangé
Quoted 72 lines · Markus Armbruster
Quoted 37 lines · Victor Toso

Hi,

Happy 1st April. Not a joke :) /* ugh, took me too long to send */

This series is about adding a generator in scripts/qapi to produce Go data structures that can be used to communicate with QEMU over QMP.

* Why Go?

There are quite a few Go projects that interact with QEMU over QMP and they endup using a mix of different libraries with their own code.

** Which projects?

The ones I've found so far:

- podman machine
https://github.com/containers/podman/tree/main/pkg/machine/qemu

- kata-containers (govmm)
https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm

- lxd
https://github.com/lxc/lxd/tree/master/lxd/instance/drivers

- kubevirt (plain json strings)
https://github.com/kubevirt/kubevirt

(let me know if you know others)

* But Why?

Quoted 8 lines

I'm particularly interested in 3 out of 4 of the projects above and only Kubevirt uses libvirt to handle QEMU. That means that every QEMU releases where a QMP command, event or other data struct is added, removed or changed, those projects need to check what changed in QEMU and then address those changes in their projects, if needed.

The idea behind generating Go data structures is that we can keep a Go module which can have releases that follow QEMU releases.

We need to look at "following the QEMU releases" a bit more closely.

Merging your patches gives us the capability to generate a Go interface to HEAD's version of QMP.

The obvious way for an out-of-tree Go program to use this generated Go interface is to build with a specific version of it. It can then talk QMP to any compatible QEMU version.

Compatibility with older QEMUs is not assured: stuff added since is present on the Go QMP client end, but not on the QEMU QMP server end.

Compatibility with newer QEMUs is subject to our deprecation policy:

In general features are intended to be supported indefinitely once introduced into QEMU. In the event that a feature needs to be removed, it will be listed in this section. The feature will remain functional for the release in which it was deprecated and one further release. After these two releases, the feature is liable to be removed.

So, if you stay away from deprecated stuff, you're good for two more releases at least.

Quoted 8 lines
Quoted 1 line

Does this work for the projects you have in mind?

It might work for some projects, but in the general case I find it pretty unappealing as a restriction. Mixing and matching new QEMU with old libvirt, or vica-verca has been an incredibly common thing todo when both developing and perhaps more importantly debugging problems. For example I have one libvirt build and I use it against any QEMU from Fedora / any RHEL-8.x update, which spans a great many QEMU releases.

I'd like to propose that for compatibility with a wide range of QEMU versions, you use or reinvent libvirt.

Quoted 11 lines

I like the idea of auto-generating clients from the QAPI schema, and would like it if we were able to use this kind of approach on the libvirt side, but for that we need to be more flexible in version matching.

Our current approach to deprecation features and subsequently removing them from the QAPI schema works fine when the QAPI schema is only used internally by QEMU, not when we we expand usage of QAPI to external applications.

I think we need to figure out a way to make the QAPI schema itself be append only, while still allowing QEMU to deprecation & remove features.

This is going to get complicated fast.

Quoted 8 lines

For a minimum viable use case, this doesn't feel all that difficult, as conceptually instead of deleting the field from QAPI, we just need to annotate it to say when it was deleted from the QEMU side. The QAPI generator for internal QEMU usage, can omit any fields annotated as deleted in QAPI schema. The QAPI generator for external app usage, can (optionally) be told to include deleted fields ranging back to a given version number. So apps can chooses what degree of compat they wish to retain.

Consider this evolution of command block_resize

* Initially, it has a mandatory argument @device[*].

* An alternative way to specify the command's object emerges: new argument @node-name. Both old @device and new @node-name become optional, and exactly one of them must be specified. This is commit 3b1dbd11a6 "qmp: Allow block_resize to manipulate bs graph nodes."

* At some future date, the old way gets deprecated: argument @device acquires feature @deprecated.

* Still later, the old way gets removed: @device is deleted, and @node-name becomes mandatory.

What is the proper version-spanning interface?

I figure it's both arguments optional, must specify the right one for the version of QEMU actually in use. This spans versions, but it fails to abstract from them.

Note that it's not enough to replace "delete member" by "mark member deleted in <version>". You also have to keep full history for "is it optional". And for types, because those can evolve compatibly, too, e.g. from struct to flat union, or from string to alternate of string and something else. What is the proper version-spanning interface in all the possible cases?

Quoted 3 lines

Apps that wish to have version compat, would of course need to write their code to be aware of which fields they need to seend for which QEMU version.

At which point we're reinventing libvirt.

Quoted 24 lines
Quoted 24 lines
Quoted 23 lines

* Status

There are a few rough edges to work on but this is usable. The major thing I forgot to add is handling Error from Commands. It'll be the first thing I'll work on next week.

If you want to start using this Today you can fetch it in at

https://gitlab.com/victortoso/qapi-go/

There are quite a few tests that I took from the examples in the qapi schema. Coverage using go's cover tool is giving 28.6% of statements

I've uploaded the a static generated godoc output of the above Go module here:

https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/

* License

While the generator (golang.py in this series) is GPL v2, the

Quoted 7 lines
Quoted 2 lines

I'd make it v2+, just to express my displeasure with the decision to make the initial QAPI generator v2 only for no good reason at all.

Our policy is that all new code should be v2+ anyway, unless it was clearly derived from some pre-existing v2-only code. Upto Victor to say whether the golang.py is considered clean, or was copy+paste in any parts from existin v2-only code

Makes sense.

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

generated code needs to be compatible with other Golang projects, such as the ones mentioned above. My intention is to keep a Go module with a MIT license.

Meh. Can't be helped, I guess.

This does make me wonder though whether the license of the QAPI input files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is the Go code to be considered a derived work of the QAPI JSON files. I'm not finding a clearly articulated POV on this question so far.

Oww. You're right.

The safe and easy answer is "same license as the generator code". Anything else is either not safe or not easy, I'm afraid.

[*] Because everyhing in QEMU must be called either "device" or "driver". It's the law!

#53 lore raw
Quoted 36 lines · Markus Armbruster
Quoted 25 lines · Daniel P. Berrangé
Quoted 24 lines · Markus Armbruster

We need to look at "following the QEMU releases" a bit more closely.

Merging your patches gives us the capability to generate a Go interface to HEAD's version of QMP.

The obvious way for an out-of-tree Go program to use this generated Go interface is to build with a specific version of it. It can then talk QMP to any compatible QEMU version.

Compatibility with older QEMUs is not assured: stuff added since is present on the Go QMP client end, but not on the QEMU QMP server end.

Compatibility with newer QEMUs is subject to our deprecation policy:

In general features are intended to be supported indefinitely once introduced into QEMU. In the event that a feature needs to be removed, it will be listed in this section. The feature will remain functional for the release in which it was deprecated and one further release. After these two releases, the feature is liable to be removed.

So, if you stay away from deprecated stuff, you're good for two more releases at least.

Quoted 8 lines
Quoted 1 line

Does this work for the projects you have in mind?

It might work for some projects, but in the general case I find it pretty unappealing as a restriction. Mixing and matching new QEMU with old libvirt, or vica-verca has been an incredibly common thing todo when both developing and perhaps more importantly debugging problems. For example I have one libvirt build and I use it against any QEMU from Fedora / any RHEL-8.x update, which spans a great many QEMU releases.

Quoted 2 lines

I'd like to propose that for compatibility with a wide range of QEMU versions, you use or reinvent libvirt.

Implicit in that statement though is that libvirt will not be able to make use of the QAPI code generator as proposed though. If we are designing something to make our application consumer's lives easier, but we exclude such a major application, is our solution actually a good one.

Quoted 10 lines
Quoted 8 lines

For a minimum viable use case, this doesn't feel all that difficult, as conceptually instead of deleting the field from QAPI, we just need to annotate it to say when it was deleted from the QEMU side. The QAPI generator for internal QEMU usage, can omit any fields annotated as deleted in QAPI schema. The QAPI generator for external app usage, can (optionally) be told to include deleted fields ranging back to a given version number. So apps can chooses what degree of compat they wish to retain.

Consider this evolution of command block_resize

To help us understand, I'll illustrate some possible interfaces in both Go and Python, since that covers dynamic and static languages

Quoted 1 line

* Initially, it has a mandatory argument @device[*].

Python definition:

def block_resize(device, size)

Caller:

block_resize('dev0', 1*GiB)

Golang definition

   type BlockResizeCommand struct {
       Device string
       Size int
   }

Caller

   cmd := &BlockResizeCommand{
       Device: "dev0",
       Size: 1 * GiB,
   }
Quoted 4 lines

* An alternative way to specify the command's object emerges: new argument @node-name. Both old @device and new @node-name become optional, and exactly one of them must be specified. This is commit 3b1dbd11a6 "qmp: Allow block_resize to manipulate bs graph nodes."

Python definition. Tricky, as non-optional params must be before optional params, but size is naturally the last arg. One option is to pointlessly mark 'size' as optional

def block_resize(device=None, node_name=None, size=None)

Caller

    block_resize(device="dev0", size=1*GiB)
    block_resize(node_name="devnode0", size=1*GiB)

In golang definition

   type BlockResizeArguments struct {
       Device string
       NodeName string
       Size int
   }

Caller choice of

   cmd := &BlockResizeCommand{
       Device: "dev0",
       Size: 1 * GiB,
   }

   cmd := &BlockResizeCommand{
       NodeName: "devnode0",
       Size: 1 * GiB,
   }

Neither case can easily prevent passing Device and NodeName at same time.

Quoted 2 lines

* At some future date, the old way gets deprecated: argument @device acquires feature @deprecated.

Ok, no change needed to the APIs in either case. Possibly have code emit a warning if a deprecated field is set.

Quoted 2 lines

* Still later, the old way gets removed: @device is deleted, and @node-name becomes mandatory.

Again no change needed to APIs, but QEMU will throw back an error if the wrong one is used.

Quoted 5 lines

What is the proper version-spanning interface?

I figure it's both arguments optional, must specify the right one for the version of QEMU actually in use. This spans versions, but it fails to abstract from them.

Yep, I think that's inevitable in this scenario. THe plus side is that apps that want to span versions can do so. The downside is that apps that don't want smarts to span version, may loose compile time warnings about use of the now deleted field.

I suggested the code generator have an option to say what level of compat to use for generated code, so that apps can request an API without compat, which will result in compile errors. This though assumes every consumer app is embedding their own generated copy of the code. Not neccessarily desirable.

At the C level you can play games with __deprecated__ to get compile time warnings in some cases.

#define GLIB_VERSION_MIN_REQUIRED GLIB_VERSION_2_56

causes QEMU to get compile time warnings (or errors) if it attempts to use a API feature deprecated in 2.56, even if the API exists in the header & library.

Quoted 6 lines

Note that it's not enough to replace "delete member" by "mark member deleted in <version>". You also have to keep full history for "is it optional". And for types, because those can evolve compatibly, too, e.g. from struct to flat union, or from string to alternate of string and something else. What is the proper version-spanning interface in all the possible cases?

I've not thought through all possible scenarios, but there may end up being restrictions, such that changes that were previously possible may have to be forbidden.

One example, in the past we could do deprecate a field 'foo', then delete 'foo' and then some time re-introduce 'foo' with a completely different type. That would not be possible if we wanted to maintain compat, but in this example that's probably a good thing, as it'll be super confusing to have the same field name change type like that over time. Easier to just use a different name.

So the question to me is not whether all our previous changes are still possible, but whether enough of the typwes of change are possible, such that we can cope with the ongoing maint in a reasonable way. I don't think we've explored the possibility enough to say one way or the other.

Quoted 5 lines
Quoted 3 lines

Apps that wish to have version compat, would of course need to write their code to be aware of which fields they need to seend for which QEMU version.

At which point we're reinventing libvirt.

The premise of the code generators is that there are applications that want to consume QEMU that are not libvirt. With this line of reasoning we could easily say that all such applications should just use libvirt and then we don't need to provide any of these code generators. The fact that we're considering these code generators though, says that we're accepting there are valid use cases that don't want to use libvirt for whatever reasons. It is reasonable that some of those applications may wish to target a wide range of QEMU versions, just like libvirt does.

It is also reasonable to say that libvirt would be better off if it could auto-generate a client API for QEMU too, instead of writing it by hand from a human reading the QAPI

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#54 lore raw
Quoted 96 lines · Daniel P. Berrangé
Quoted 10 lines · Markus Armbruster
Quoted 8 lines

For a minimum viable use case, this doesn't feel all that difficult, as conceptually instead of deleting the field from QAPI, we just need to annotate it to say when it was deleted from the QEMU side. The QAPI generator for internal QEMU usage, can omit any fields annotated as deleted in QAPI schema. The QAPI generator for external app usage, can (optionally) be told to include deleted fields ranging back to a given version number. So apps can chooses what degree of compat they wish to retain.

Consider this evolution of command block_resize

To help us understand, I'll illustrate some possible interfaces in both Go and Python, since that covers dynamic and static languages

Quoted 1 line

* Initially, it has a mandatory argument @device[*].

Python definition:

def block_resize(device, size)

Caller:

block_resize('dev0', 1*GiB)

Golang definition

   type BlockResizeCommand struct {
       Device string
       Size int
   }

Caller

   cmd := &BlockResizeCommand{
       Device: "dev0",
       Size: 1 * GiB,
   }
Quoted 4 lines

* An alternative way to specify the command's object emerges: new argument @node-name. Both old @device and new @node-name become optional, and exactly one of them must be specified. This is commit 3b1dbd11a6 "qmp: Allow block_resize to manipulate bs graph nodes."

Python definition. Tricky, as non-optional params must be before optional params, but size is naturally the last arg. One option is to pointlessly mark 'size' as optional

def block_resize(device=None, node_name=None, size=None)

Caller

    block_resize(device="dev0", size=1*GiB)
    block_resize(node_name="devnode0", size=1*GiB)

In golang definition

   type BlockResizeArguments struct {
       Device string
       NodeName string
       Size int
   }

Caller choice of

   cmd := &BlockResizeCommand{
       Device: "dev0",
       Size: 1 * GiB,
   }

   cmd := &BlockResizeCommand{
       NodeName: "devnode0",
       Size: 1 * GiB,
   }

Neither case can easily prevent passing Device and NodeName at same time.

Quoted 2 lines

* At some future date, the old way gets deprecated: argument @device acquires feature @deprecated.

Ok, no change needed to the APIs in either case. Possibly have code emit a warning if a deprecated field is set.

Quoted 2 lines

* Still later, the old way gets removed: @device is deleted, and @node-name becomes mandatory.

Again no change needed to APIs, but QEMU will throw back an error if the wrong one is used.

Quoted 2 lines

What is the proper version-spanning interface?

Quoted 8 lines
Quoted 3 lines

I figure it's both arguments optional, must specify the right one for the version of QEMU actually in use. This spans versions, but it fails to abstract from them.

Yep, I think that's inevitable in this scenario. THe plus side is that apps that want to span versions can do so. The downside is that apps that don't want smarts to span version, may loose compile time warnings about use of the now deleted field.

Having said that, a different way to approach the problem is to expose the versioning directly in the generated code.

Consider a QAPI with versioning info about the fields

{ 'command': 'block_resize',

    'since': '5.0.0',
    'data': { 'device': ['type': 'str', 'until': '5.2.0' ],
              '*device': ['type': 'str', 'since': '5.2.0', 'until': '7.0.0' ],
              '*node-name': ['type': 'str', 'since': '5.2.0', 'until: '7.0.0' ],
              'node-name': ['type': 'str', 'since': '7.0.0' ],
              'size': 'int' } }

Meaning

* Introduced in 5.0.0, with 'device' mandatory
* In 5.2.0, 'device' becomes optional, with optional 'node-name' as alternative
* In 7.0.0, 'device' is deleted, and 'node-name' becomes mandatory

Now consider the Go structs

In 5.0.0 we can generate:

   type BlockResizeArguments struct {
       V500 *BlockResizeArguments500
   }

   type BlockResizeArgumentsV1 struct {
        Device string
        Size int
    }

app can use

    dev := "dev0"
    cmd := BlockResizeArguments{
       V500: &BlockResizeArguments500{
          Device: dev,
	  Size: 1 * GiB
       }
    }

In 5.2.0 we can now generate

   type BlockResizeArguments struct {
       V500 *BlockResizeArgumentsV500
       V520 *BlockResizeArgumentsV520
   }

   type BlockResizeArgumentsV500 struct {
        Device string
        Size int
    }
   type BlockResizeArgumentsV520 struct {
        Device *string
	NodeName *string
        Size int
    }

App can use the same as before, or switch to one of

    dev := "dev0"
    cmd := BlockResizeArguments{
       V520: &BlockResizeArguments520{
          Device: &dev,
	  Size: 1 * GiB
       }
    }

or

    node := "nodedev0"
    cmd := BlockResizeArguments{
       V520: &BlockResizeArguments520{
          NodeName: &node,
	  Size: 1 * GiB
       }
    }

In 7.0.0 we can now generate

   type BlockResizeArguments struct {
       V500 *BlockResizeArgumentsV500
       V520 *BlockResizeArgumentsV520
       V700 *BlockResizeArgumentsV700
   }

   type BlockResizeArgumentsV500 struct {
        Device string
        Size int
   }

type BlockResizeArgumentsV520 struct {
Device *string
NodeName *string
Size int
}

type BlockResizeArgumentsV700 struct {
NodeName string
Size int
}

App can use the same as before, or switch to

    node := "nodedev0"
    cmd := BlockResizeArguments{
       V700: &BlockResizeArguments700{
          NodeName: node,
	  Size: 1 * GiB
       }
    }

This kind of per-command/type versioning is not uncommon when defining API protocols/interfaces.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#55 lore raw
Quoted 228 lines · Daniel P. Berrangé
Quoted 96 lines · Daniel P. Berrangé
Quoted 10 lines · Markus Armbruster
Quoted 8 lines

For a minimum viable use case, this doesn't feel all that difficult, as conceptually instead of deleting the field from QAPI, we just need to annotate it to say when it was deleted from the QEMU side. The QAPI generator for internal QEMU usage, can omit any fields annotated as deleted in QAPI schema. The QAPI generator for external app usage, can (optionally) be told to include deleted fields ranging back to a given version number. So apps can chooses what degree of compat they wish to retain.

Consider this evolution of command block_resize

To help us understand, I'll illustrate some possible interfaces in both Go and Python, since that covers dynamic and static languages

Quoted 1 line

* Initially, it has a mandatory argument @device[*].

Python definition:

def block_resize(device, size)

Caller:

block_resize('dev0', 1*GiB)

Golang definition

   type BlockResizeCommand struct {
       Device string
       Size int
   }

Caller

   cmd := &BlockResizeCommand{
       Device: "dev0",
       Size: 1 * GiB,
   }
Quoted 4 lines

* An alternative way to specify the command's object emerges: new argument @node-name. Both old @device and new @node-name become optional, and exactly one of them must be specified. This is commit 3b1dbd11a6 "qmp: Allow block_resize to manipulate bs graph nodes."

Python definition. Tricky, as non-optional params must be before optional params, but size is naturally the last arg. One option is to pointlessly mark 'size' as optional

def block_resize(device=None, node_name=None, size=None)

Caller

    block_resize(device="dev0", size=1*GiB)
    block_resize(node_name="devnode0", size=1*GiB)

In golang definition

   type BlockResizeArguments struct {
       Device string
       NodeName string
       Size int
   }

Caller choice of

   cmd := &BlockResizeCommand{
       Device: "dev0",
       Size: 1 * GiB,
   }

   cmd := &BlockResizeCommand{
       NodeName: "devnode0",
       Size: 1 * GiB,
   }

Neither case can easily prevent passing Device and NodeName at same time.

Quoted 2 lines

* At some future date, the old way gets deprecated: argument @device acquires feature @deprecated.

Ok, no change needed to the APIs in either case. Possibly have code emit a warning if a deprecated field is set.

Quoted 2 lines

* Still later, the old way gets removed: @device is deleted, and @node-name becomes mandatory.

Again no change needed to APIs, but QEMU will throw back an error if the wrong one is used.

Quoted 2 lines

What is the proper version-spanning interface?

Quoted 8 lines
Quoted 3 lines

I figure it's both arguments optional, must specify the right one for the version of QEMU actually in use. This spans versions, but it fails to abstract from them.

Yep, I think that's inevitable in this scenario. THe plus side is that apps that want to span versions can do so. The downside is that apps that don't want smarts to span version, may loose compile time warnings about use of the now deleted field.

Having said that, a different way to approach the problem is to expose the versioning directly in the generated code.

Consider a QAPI with versioning info about the fields

{ 'command': 'block_resize',

    'since': '5.0.0',
    'data': { 'device': ['type': 'str', 'until': '5.2.0' ],
              '*device': ['type': 'str', 'since': '5.2.0', 'until': '7.0.0' ],
              '*node-name': ['type': 'str', 'since': '5.2.0', 'until: '7.0.0' ],
              'node-name': ['type': 'str', 'since': '7.0.0' ],
              'size': 'int' } }

Meaning

* Introduced in 5.0.0, with 'device' mandatory
* In 5.2.0, 'device' becomes optional, with optional 'node-name' as alternative
* In 7.0.0, 'device' is deleted, and 'node-name' becomes mandatory

Now consider the Go structs

In 5.0.0 we can generate:

   type BlockResizeArguments struct {
       V500 *BlockResizeArguments500
   }

   type BlockResizeArgumentsV1 struct {
        Device string
        Size int
    }

app can use

    dev := "dev0"
    cmd := BlockResizeArguments{
       V500: &BlockResizeArguments500{
          Device: dev,
	  Size: 1 * GiB
       }
    }

In 5.2.0 we can now generate

   type BlockResizeArguments struct {
       V500 *BlockResizeArgumentsV500
       V520 *BlockResizeArgumentsV520
   }

   type BlockResizeArgumentsV500 struct {
        Device string
        Size int
    }
   type BlockResizeArgumentsV520 struct {
        Device *string
	NodeName *string
        Size int
    }

App can use the same as before, or switch to one of

    dev := "dev0"
    cmd := BlockResizeArguments{
       V520: &BlockResizeArguments520{
          Device: &dev,
	  Size: 1 * GiB
       }
    }

or

    node := "nodedev0"
    cmd := BlockResizeArguments{
       V520: &BlockResizeArguments520{
          NodeName: &node,
	  Size: 1 * GiB
       }
    }

In 7.0.0 we can now generate

   type BlockResizeArguments struct {
       V500 *BlockResizeArgumentsV500
       V520 *BlockResizeArgumentsV520
       V700 *BlockResizeArgumentsV700
   }

   type BlockResizeArgumentsV500 struct {
        Device string
        Size int
   }
   type BlockResizeArgumentsV520 struct {
        Device *string
	NodeName *string
        Size int
   }

   type BlockResizeArgumentsV700 struct {
	NodeName string
        Size int
   }

App can use the same as before, or switch to

    node := "nodedev0"
    cmd := BlockResizeArguments{
       V700: &BlockResizeArguments700{
          NodeName: node,
	  Size: 1 * GiB
       }
    }
Quoted 2 lines

This kind of per-command/type versioning is not uncommon when defining API protocols/interfaces.

1. At every release, put a copy of the QAPI schema in the freezer.

2. For every copy, generate Go types with a suitable name suffix. Collect all the name stems.

3. For each name stem, define a Go struct that contains all the suffixed Go types with that stem.

Look Ma, no cluttering the QAPI schema with a full history! Also no complicating the schema language to provide the means for that.

#56 lore raw
Quoted 130 lines · Markus Armbruster
Quoted 123 lines · Daniel P. Berrangé

On Tue, May 10, 2022 at 01:34:03PM +0100, Daniel P. Berrangé wrote:
Having said that, a different way to approach the problem is to expose the versioning directly in the generated code.

Consider a QAPI with versioning info about the fields

{ 'command': 'block_resize',

    'since': '5.0.0',
    'data': { 'device': ['type': 'str', 'until': '5.2.0' ],
              '*device': ['type': 'str', 'since': '5.2.0', 'until': '7.0.0' ],
              '*node-name': ['type': 'str', 'since': '5.2.0', 'until: '7.0.0' ],
              'node-name': ['type': 'str', 'since': '7.0.0' ],
              'size': 'int' } }

Meaning

* Introduced in 5.0.0, with 'device' mandatory
* In 5.2.0, 'device' becomes optional, with optional 'node-name' as alternative
* In 7.0.0, 'device' is deleted, and 'node-name' becomes mandatory

Now consider the Go structs

In 5.0.0 we can generate:

   type BlockResizeArguments struct {
       V500 *BlockResizeArguments500
   }

   type BlockResizeArgumentsV1 struct {
        Device string
        Size int
    }

app can use

    dev := "dev0"
    cmd := BlockResizeArguments{
       V500: &BlockResizeArguments500{
          Device: dev,
	  Size: 1 * GiB
       }
    }

In 5.2.0 we can now generate

   type BlockResizeArguments struct {
       V500 *BlockResizeArgumentsV500
       V520 *BlockResizeArgumentsV520
   }

   type BlockResizeArgumentsV500 struct {
        Device string
        Size int
    }
   type BlockResizeArgumentsV520 struct {
        Device *string
	NodeName *string
        Size int
    }

App can use the same as before, or switch to one of

    dev := "dev0"
    cmd := BlockResizeArguments{
       V520: &BlockResizeArguments520{
          Device: &dev,
	  Size: 1 * GiB
       }
    }

or

    node := "nodedev0"
    cmd := BlockResizeArguments{
       V520: &BlockResizeArguments520{
          NodeName: &node,
	  Size: 1 * GiB
       }
    }

In 7.0.0 we can now generate

   type BlockResizeArguments struct {
       V500 *BlockResizeArgumentsV500
       V520 *BlockResizeArgumentsV520
       V700 *BlockResizeArgumentsV700
   }

   type BlockResizeArgumentsV500 struct {
        Device string
        Size int
   }
   type BlockResizeArgumentsV520 struct {
        Device *string
	NodeName *string
        Size int
   }

   type BlockResizeArgumentsV700 struct {
	NodeName string
        Size int
   }

App can use the same as before, or switch to

    node := "nodedev0"
    cmd := BlockResizeArguments{
       V700: &BlockResizeArguments700{
          NodeName: node,
	  Size: 1 * GiB
       }
    }
Quoted 2 lines

This kind of per-command/type versioning is not uncommon when defining API protocols/interfaces.

1. At every release, put a copy of the QAPI schema in the freezer.

Quoted 8 lines

2. For every copy, generate Go types with a suitable name suffix. Collect all the name stems.

3. For each name stem, define a Go struct that contains all the suffixed Go types with that stem.

Look Ma, no cluttering the QAPI schema with a full history! Also no complicating the schema language to provide the means for that.

That could indeed be a viable approach. Puts a little more work on the code generator to match up the types, but probably isn't too hard.

Incidentally, we've intentionally not exposed type names in the QAPI introspection in QMP. With code generators, when the generated code type names driven from QAPI schema, there's likely going to be an expectation that type names in QAPI have some kind of stability rules.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#57 lore raw
In reply to Daniel P. Berrangé
Quoted 26 lines · Daniel P. Berrangé

In 7.0.0 we can now generate

   type BlockResizeArguments struct {
       V500 *BlockResizeArgumentsV500
       V520 *BlockResizeArgumentsV520
       V700 *BlockResizeArgumentsV700
   }

   type BlockResizeArgumentsV500 struct {
       Device string
       Size int
   }

   type BlockResizeArgumentsV520 struct {
       Device *string
       NodeName *string
       Size int
   }

   type BlockResizeArgumentsV700 struct {
       NodeName string
       Size int
   }

App can use the same as before, or switch to

Quoted 7 lines
    node := "nodedev0"
    cmd := BlockResizeArguments{
        V700: &BlockResizeArguments700{
            NodeName: node,
            Size: 1 * GiB
        }
    }

This honestly looks pretty unwieldy.

If the application already knows it's targeting a specific version of the QEMU API, which for the above code to make any sense it will have to, couldn't it do something like

import qemu .../qemu/v700

at the beginning of the file and then use regular old

  cmd := qemu.BlockResizeArguments{
      NodeName: nodeName,
      Size: size,
  }

instead?

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
#58 lore raw
Quoted 35 lines · Andrea Bolognani
Quoted 26 lines · Daniel P. Berrangé

In 7.0.0 we can now generate

   type BlockResizeArguments struct {
       V500 *BlockResizeArgumentsV500
       V520 *BlockResizeArgumentsV520
       V700 *BlockResizeArgumentsV700
   }

   type BlockResizeArgumentsV500 struct {
       Device string
       Size int
   }

   type BlockResizeArgumentsV520 struct {
       Device *string
       NodeName *string
       Size int
   }

   type BlockResizeArgumentsV700 struct {
       NodeName string
       Size int
   }

App can use the same as before, or switch to

Quoted 7 lines
    node := "nodedev0"
    cmd := BlockResizeArguments{
        V700: &BlockResizeArguments700{
            NodeName: node,
            Size: 1 * GiB
        }
    }
Quoted 1 line

This honestly looks pretty unwieldy.

It isn't all that more verbose than without the versions - just a single struct wrapper.

Quoted 7 lines

If the application already knows it's targeting a specific version of the QEMU API, which for the above code to make any sense it will have to, couldn't it do something like

import qemu .../qemu/v700

Quoted 8 lines

at the beginning of the file and then use regular old

  cmd := qemu.BlockResizeArguments{
      NodeName: nodeName,
      Size: size,
  }

instead?

This would lead to a situation where every struct is duplicated for every version, even though 90% of the time they'll be identical across multiple versions. This is not very ammenable to the desire to be able to dynamically choose per-command which version you want based on which version of QEMU you're connected to.

ie

     var cmd Command
     if qmp.HasVersion(qemu.Version(7, 0, 0)) {
        cmd = BlockResizeArguments{
           V700: &BlockResizeArguments700{
             NodeName: node,
             Size: 1 * GiB
	   }
         }
     } else {
        cmd = BlockResizeArguments{
           V520: &BlockResizeArguments520{
             Device: dev,
             Size: 1 * GiB
	   }
         }
     }

And of course the HasVersion check is going to be different for each command that matters.

Having said that, this perhaps shows the nested structs are overkill. We could have

     var cmd Command
     if qmp.HasVersion(qemu.Version(7, 0, 0)) {
         cmd = &BlockResizeArguments700{
             NodeName: node,
             Size: 1 * GiB
         }
     } else {
        cmd = &BlockResizeArguments520{
             Device: dev,
             Size: 1 * GiB
         }
     }

If there was some need for common handling of the different versioned variants, we could still have a 'BlockResizeArguments' that has a field per version, as an optional thing. Or have a BlockResizeArguments interface, implemented by each version </hand-wavey>

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#59 lore raw
Quoted 31 lines · Daniel P. Berrangé

This would lead to a situation where every struct is duplicated for every version, even though 90% of the time they'll be identical across multiple versions. This is not very ammenable to the desire to be able to dynamically choose per-command which version you want based on which version of QEMU you're connected to.

ie

     var cmd Command
     if qmp.HasVersion(qemu.Version(7, 0, 0)) {
        cmd = BlockResizeArguments{
           V700: &BlockResizeArguments700{
             NodeName: node,
             Size: 1 * GiB
	   }
         }
     } else {
        cmd = BlockResizeArguments{
           V520: &BlockResizeArguments520{
             Device: dev,
             Size: 1 * GiB
	   }
         }
     }

And of course the HasVersion check is going to be different for each command that matters.

Having said that, this perhaps shows the nested structs are overkill. We could have

Quoted 12 lines
     var cmd Command
     if qmp.HasVersion(qemu.Version(7, 0, 0)) {
         cmd = &BlockResizeArguments700{
             NodeName: node,
             Size: 1 * GiB
         }
     } else {
        cmd = &BlockResizeArguments520{
             Device: dev,
             Size: 1 * GiB
         }
     }

Right, making the decision at the import level would require adding a level of indirection and make this kind of dynamic logic awkward.

We shouldn't force users to sprinkle version numbers everywhere though, especially since different commands will change at different points in time. It should be possible to do something like

  if !qmp.HasAPI(600) {
      panic("update QEMU")
  }

  cmd := &BlockResizeArguments600{ // really BlockResizeArguments520
      Device: device,
      Size:   size,
  }

or even

  if !qmp.HasAPI(qmp.API.Latest) {
      panic("update QEMU")
  }

  cmd := &BlockResizeArguments{
      NodeName: nodeName,
      Size:     size,
  }

Should be easy enough to achieve with type aliases.

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
#60 lore raw
Quoted 72 lines · Andrea Bolognani
Quoted 31 lines · Daniel P. Berrangé

This would lead to a situation where every struct is duplicated for every version, even though 90% of the time they'll be identical across multiple versions. This is not very ammenable to the desire to be able to dynamically choose per-command which version you want based on which version of QEMU you're connected to.

ie

     var cmd Command
     if qmp.HasVersion(qemu.Version(7, 0, 0)) {
        cmd = BlockResizeArguments{
           V700: &BlockResizeArguments700{
             NodeName: node,
             Size: 1 * GiB
	   }
         }
     } else {
        cmd = BlockResizeArguments{
           V520: &BlockResizeArguments520{
             Device: dev,
             Size: 1 * GiB
	   }
         }
     }

And of course the HasVersion check is going to be different for each command that matters.

Having said that, this perhaps shows the nested structs are overkill. We could have

Quoted 12 lines
     var cmd Command
     if qmp.HasVersion(qemu.Version(7, 0, 0)) {
         cmd = &BlockResizeArguments700{
             NodeName: node,
             Size: 1 * GiB
         }
     } else {
        cmd = &BlockResizeArguments520{
             Device: dev,
             Size: 1 * GiB
         }
     }

Right, making the decision at the import level would require adding a level of indirection and make this kind of dynamic logic awkward.

We shouldn't force users to sprinkle version numbers everywhere though, especially since different commands will change at different points in time. It should be possible to do something like

  if !qmp.HasAPI(600) {
      panic("update QEMU")
  }

  cmd := &BlockResizeArguments600{ // really BlockResizeArguments520
      Device: device,
      Size:   size,
  }

or even

  if !qmp.HasAPI(qmp.API.Latest) {
      panic("update QEMU")
  }

  cmd := &BlockResizeArguments{
      NodeName: nodeName,
      Size:     size,
  }
Quoted 1 line

Should be easy enough to achieve with type aliases.

I guess we would have a single package which does

typedef BlockResizeArguments520 BlockResizeArguments; ...for each type...

The interaction with API versioning will be tedious though. For the versioned structs we'll be forever back compatible, due to the append-only nature of the versioned struct approach. For the type aliases, we'll be forever breaking compat as at least 1 struct alias is likely to need changing every QEMU release.

Might suggest having 2 distinct go modules. A base module which is versioned as a stable API with semver (forever v1.xxx), and an add-on module that depends on base module, which is versioned as an unstable API with semver (forever v0.xxx)

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#61 lore raw
In reply to Daniel P. Berrangé

Hi,

Quoted 90 lines · Daniel P. Berrangé
Quoted 35 lines · Andrea Bolognani
Quoted 26 lines · Daniel P. Berrangé

In 7.0.0 we can now generate

   type BlockResizeArguments struct {
       V500 *BlockResizeArgumentsV500
       V520 *BlockResizeArgumentsV520
       V700 *BlockResizeArgumentsV700
   }

   type BlockResizeArgumentsV500 struct {
       Device string
       Size int
   }

   type BlockResizeArgumentsV520 struct {
       Device *string
       NodeName *string
       Size int
   }

   type BlockResizeArgumentsV700 struct {
       NodeName string
       Size int
   }

App can use the same as before, or switch to

Quoted 7 lines
    node := "nodedev0"
    cmd := BlockResizeArguments{
        V700: &BlockResizeArguments700{
            NodeName: node,
            Size: 1 * GiB
        }
    }
Quoted 1 line

This honestly looks pretty unwieldy.

It isn't all that more verbose than without the versions - just a single struct wrapper.

Quoted 7 lines

If the application already knows it's targeting a specific version of the QEMU API, which for the above code to make any sense it will have to, couldn't it do something like

import qemu .../qemu/v700

Quoted 8 lines

at the beginning of the file and then use regular old

  cmd := qemu.BlockResizeArguments{
      NodeName: nodeName,
      Size: size,
  }

instead?

This would lead to a situation where every struct is duplicated for every version, even though 90% of the time they'll be identical across multiple versions. This is not very ammenable to the desire to be able to dynamically choose per-command which version you want based on which version of QEMU you're connected to.

ie

     var cmd Command
     if qmp.HasVersion(qemu.Version(7, 0, 0)) {
        cmd = BlockResizeArguments{
           V700: &BlockResizeArguments700{
             NodeName: node,
             Size: 1 * GiB
	   }
         }
     } else {
        cmd = BlockResizeArguments{
           V520: &BlockResizeArguments520{
             Device: dev,
             Size: 1 * GiB
	   }
         }
     }

And of course the HasVersion check is going to be different for each command that matters.

Having said that, this perhaps shows the nested structs are overkill. We could have

Quoted 12 lines
     var cmd Command
     if qmp.HasVersion(qemu.Version(7, 0, 0)) {
         cmd = &BlockResizeArguments700{
             NodeName: node,
             Size: 1 * GiB
         }
     } else {
        cmd = &BlockResizeArguments520{
             Device: dev,
             Size: 1 * GiB
         }
     }

The else block would be wrong in versions above 7.0.0 where block_resize changed. There will be a need to know for a specific Type if we are covered with latest qemu/qapi-go or not. Not yet sure how to address that, likely we will need to keep the information that something has been added/changed/removed per version per Type in qapi-go...

Still, I think the above proposal is a good compromise to make..

Quoted 4 lines

If there was some need for common handling of the different versioned variants, we could still have a 'BlockResizeArguments' that has a field per version, as an optional thing. Or have a BlockResizeArguments interface, implemented by each version </hand-wavey>

Cheers,
Victor

#62 lore raw
Quoted 106 lines · Victor Toso

Hi,

Quoted 90 lines · Daniel P. Berrangé
Quoted 35 lines · Andrea Bolognani
Quoted 26 lines · Daniel P. Berrangé

In 7.0.0 we can now generate

   type BlockResizeArguments struct {
       V500 *BlockResizeArgumentsV500
       V520 *BlockResizeArgumentsV520
       V700 *BlockResizeArgumentsV700
   }

   type BlockResizeArgumentsV500 struct {
       Device string
       Size int
   }

   type BlockResizeArgumentsV520 struct {
       Device *string
       NodeName *string
       Size int
   }

   type BlockResizeArgumentsV700 struct {
       NodeName string
       Size int
   }

App can use the same as before, or switch to

Quoted 7 lines
    node := "nodedev0"
    cmd := BlockResizeArguments{
        V700: &BlockResizeArguments700{
            NodeName: node,
            Size: 1 * GiB
        }
    }
Quoted 1 line

This honestly looks pretty unwieldy.

It isn't all that more verbose than without the versions - just a single struct wrapper.

Quoted 7 lines

If the application already knows it's targeting a specific version of the QEMU API, which for the above code to make any sense it will have to, couldn't it do something like

import qemu .../qemu/v700

Quoted 8 lines

at the beginning of the file and then use regular old

  cmd := qemu.BlockResizeArguments{
      NodeName: nodeName,
      Size: size,
  }

instead?

This would lead to a situation where every struct is duplicated for every version, even though 90% of the time they'll be identical across multiple versions. This is not very ammenable to the desire to be able to dynamically choose per-command which version you want based on which version of QEMU you're connected to.

ie

     var cmd Command
     if qmp.HasVersion(qemu.Version(7, 0, 0)) {
        cmd = BlockResizeArguments{
           V700: &BlockResizeArguments700{
             NodeName: node,
             Size: 1 * GiB
	   }
         }
     } else {
        cmd = BlockResizeArguments{
           V520: &BlockResizeArguments520{
             Device: dev,
             Size: 1 * GiB
	   }
         }
     }

And of course the HasVersion check is going to be different for each command that matters.

Having said that, this perhaps shows the nested structs are overkill. We could have

Quoted 12 lines
     var cmd Command
     if qmp.HasVersion(qemu.Version(7, 0, 0)) {
         cmd = &BlockResizeArguments700{
             NodeName: node,
             Size: 1 * GiB
         }
     } else {
        cmd = &BlockResizeArguments520{
             Device: dev,
             Size: 1 * GiB
         }
     }
Quoted 6 lines

The else block would be wrong in versions above 7.0.0 where block_resize changed. There will be a need to know for a specific Type if we are covered with latest qemu/qapi-go or not. Not yet sure how to address that, likely we will need to keep the information that something has been added/changed/removed per version per Type in qapi-go...

I put that in the "nice to have" category. No application can predict the future, and nor do they really need to try in general.

If the application code was written when the newest QEMU was 7.1.0, and the above code is correct for QEMU <= 7.1.0, then that's good enough. If the BlockResizeArguments struct changed in a later QEMU version 8.0.0, that doesn't matter at the point the app code was written.

Much of the time the changes are back compatible, ie just adding a new field, and so everything will still work fine if the app carries on using BlockResizeArguments700, despite a new BlockResizeArguments800 arriving with a new field.

Only in the cases where a field was removed or changed in a non-compatible manner would an app have problems, and QEMU will happily report an error at runtime if the app sends something incompatible.

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#63 lore raw

HI,

Quoted 118 lines · Daniel P. Berrangé
Quoted 106 lines · Victor Toso

Hi,

Quoted 90 lines · Daniel P. Berrangé
Quoted 35 lines · Andrea Bolognani
Quoted 26 lines · Daniel P. Berrangé

In 7.0.0 we can now generate

   type BlockResizeArguments struct {
       V500 *BlockResizeArgumentsV500
       V520 *BlockResizeArgumentsV520
       V700 *BlockResizeArgumentsV700
   }

   type BlockResizeArgumentsV500 struct {
       Device string
       Size int
   }

   type BlockResizeArgumentsV520 struct {
       Device *string
       NodeName *string
       Size int
   }

   type BlockResizeArgumentsV700 struct {
       NodeName string
       Size int
   }

App can use the same as before, or switch to

Quoted 7 lines
    node := "nodedev0"
    cmd := BlockResizeArguments{
        V700: &BlockResizeArguments700{
            NodeName: node,
            Size: 1 * GiB
        }
    }
Quoted 1 line

This honestly looks pretty unwieldy.

It isn't all that more verbose than without the versions - just a single struct wrapper.

Quoted 7 lines

If the application already knows it's targeting a specific version of the QEMU API, which for the above code to make any sense it will have to, couldn't it do something like

import qemu .../qemu/v700

Quoted 8 lines

at the beginning of the file and then use regular old

  cmd := qemu.BlockResizeArguments{
      NodeName: nodeName,
      Size: size,
  }

instead?

This would lead to a situation where every struct is duplicated for every version, even though 90% of the time they'll be identical across multiple versions. This is not very ammenable to the desire to be able to dynamically choose per-command which version you want based on which version of QEMU you're connected to.

ie

     var cmd Command
     if qmp.HasVersion(qemu.Version(7, 0, 0)) {
        cmd = BlockResizeArguments{
           V700: &BlockResizeArguments700{
             NodeName: node,
             Size: 1 * GiB
	   }
         }
     } else {
        cmd = BlockResizeArguments{
           V520: &BlockResizeArguments520{
             Device: dev,
             Size: 1 * GiB
	   }
         }
     }

And of course the HasVersion check is going to be different for each command that matters.

Having said that, this perhaps shows the nested structs are overkill. We could have

Quoted 12 lines
     var cmd Command
     if qmp.HasVersion(qemu.Version(7, 0, 0)) {
         cmd = &BlockResizeArguments700{
             NodeName: node,
             Size: 1 * GiB
         }
     } else {
        cmd = &BlockResizeArguments520{
             Device: dev,
             Size: 1 * GiB
         }
     }
Quoted 6 lines

The else block would be wrong in versions above 7.0.0 where block_resize changed. There will be a need to know for a specific Type if we are covered with latest qemu/qapi-go or not. Not yet sure how to address that, likely we will need to keep the information that something has been added/changed/removed per version per Type in qapi-go...

I put that in the "nice to have" category. No application can predict the future, and nor do they really need to try in general.

Quoted 5 lines

If the application code was written when the newest QEMU was 7.1.0, and the above code is correct for QEMU <= 7.1.0, then that's good enough. If the BlockResizeArguments struct changed in a later QEMU version 8.0.0, that doesn't matter at the point the app code was written.

I'm not thinking at runtime, I'm thinking at compile time.

I update $myproject's qpai-go to 8.0.0 and get a warnings that some types would not work with 8.0.0 (e.g: because there is a new BlockResizeArguments800)

Quoted 9 lines

Much of the time the changes are back compatible, ie just adding a new field, and so everything will still work fine if the app carries on using BlockResizeArguments700, despite a new BlockResizeArguments800 arriving with a new field.

Only in the cases where a field was removed or changed in a non-compatible manner would an app have problems, and QEMU will happily report an error at runtime if the app sends something incompatible.

Yeah, runtime error is fine but if we can provide some extra checks at the time someone wants to move qapi-go from 7.2.0 to 8.0.0, that would be great.

Cheers,
Victor

#64 lore raw
In reply to Daniel P. Berrangé
Quoted 37 lines · Daniel P. Berrangé
Quoted 36 lines · Markus Armbruster
Quoted 33 lines · Daniel P. Berrangé
Quoted 21 lines · Markus Armbruster

We need to look at "following the QEMU releases" a bit more closely.

Merging your patches gives us the capability to generate a Go interface to HEAD's version of QMP.

The obvious way for an out-of-tree Go program to use this generated Go interface is to build with a specific version of it. It can then talk QMP to any compatible QEMU version.

Compatibility with older QEMUs is not assured: stuff added since is present on the Go QMP client end, but not on the QEMU QMP server end.

Compatibility with newer QEMUs is subject to our deprecation policy:

In general features are intended to be supported indefinitely once introduced into QEMU. In the event that a feature needs to be removed, it will be listed in this section. The feature will remain functional for the release in which it was deprecated and one further release. After these two releases, the feature is liable to be removed.

Quoted 4 lines

So, if you stay away from deprecated stuff, you're good for two more releases at least.

Does this work for the projects you have in mind?

It might work for some projects, but in the general case I find it pretty unappealing as a restriction. Mixing and matching new QEMU with old libvirt, or vica-verca has been an incredibly common thing todo when both developing and perhaps more importantly debugging problems. For example I have one libvirt build and I use it against any QEMU from Fedora / any RHEL-8.x update, which spans a great many QEMU releases.

Quoted 8 lines
Quoted 2 lines

I'd like to propose that for compatibility with a wide range of QEMU versions, you use or reinvent libvirt.

Implicit in that statement though is that libvirt will not be able to make use of the QAPI code generator as proposed though. If we are designing something to make our application consumer's lives easier, but we exclude such a major application, is our solution actually a good one.

A solution for a narrow problem we can actually deliver and maintain is better than a solution for a wider problem we can't.

Quoted 45 lines
Quoted 10 lines
Quoted 8 lines

For a minimum viable use case, this doesn't feel all that difficult, as conceptually instead of deleting the field from QAPI, we just need to annotate it to say when it was deleted from the QEMU side. The QAPI generator for internal QEMU usage, can omit any fields annotated as deleted in QAPI schema. The QAPI generator for external app usage, can (optionally) be told to include deleted fields ranging back to a given version number. So apps can chooses what degree of compat they wish to retain.

Consider this evolution of command block_resize

To help us understand, I'll illustrate some possible interfaces in both Go and Python, since that covers dynamic and static languages

Quoted 1 line

* Initially, it has a mandatory argument @device[*].

Python definition:

def block_resize(device, size)

Caller:

block_resize('dev0', 1*GiB)

Golang definition

   type BlockResizeCommand struct {
       Device string
       Size int
   }

Caller

   cmd := &BlockResizeCommand{
       Device: "dev0",
       Size: 1 * GiB,
   }
Quoted 4 lines

* An alternative way to specify the command's object emerges: new argument @node-name. Both old @device and new @node-name become optional, and exactly one of them must be specified. This is commit 3b1dbd11a6 "qmp: Allow block_resize to manipulate bs graph nodes."

Quoted 5 lines

Python definition. Tricky, as non-optional params must be before optional params, but size is naturally the last arg. One option is to pointlessly mark 'size' as optional

def block_resize(device=None, node_name=None, size=None)

Who needs compile-time checking anyway.

Back to serious. Keyword arguments might be a better match for Python bindings.

Quoted 16 lines

Caller

    block_resize(device="dev0", size=1*GiB)
    block_resize(node_name="devnode0", size=1*GiB)

In golang definition

   type BlockResizeArguments struct {
       Device string
       NodeName string
       Size int
   }

Caller choice of

Quoted 9 lines
   cmd := &BlockResizeCommand{
       Device: "dev0",
       Size: 1 * GiB,
   }

   cmd := &BlockResizeCommand{
       NodeName: "devnode0",
       Size: 1 * GiB,
   }

Note that the Go bindings you sketched effectively use (poor man's) keyword arguments.

Quoted 2 lines

Neither case can easily prevent passing Device and NodeName at same time.

That defect lies at the schema's feet.

Quoted 14 lines
Quoted 2 lines

* At some future date, the old way gets deprecated: argument @device acquires feature @deprecated.

Ok, no change needed to the APIs in either case. Possibly have code emit a warning if a deprecated field is set.

Quoted 2 lines

* Still later, the old way gets removed: @device is deleted, and @node-name becomes mandatory.

Again no change needed to APIs, but QEMU will throw back an error if the wrong one is used.

Quoted 2 lines

What is the proper version-spanning interface?

Quoted 8 lines
Quoted 3 lines

I figure it's both arguments optional, must specify the right one for the version of QEMU actually in use. This spans versions, but it fails to abstract from them.

Yep, I think that's inevitable in this scenario. THe plus side is that apps that want to span versions can do so. The downside is that apps that don't want smarts to span version, may loose compile time warnings about use of the now deleted field.

The version-spanning interface will arguably be a bad interface for any version.

Quoted 23 lines

I suggested the code generator have an option to say what level of compat to use for generated code, so that apps can request an API without compat, which will result in compile errors. This though assumes every consumer app is embedding their own generated copy of the code. Not neccessarily desirable.

At the C level you can play games with __deprecated__ to get compile time warnings in some cases.

#define GLIB_VERSION_MIN_REQUIRED GLIB_VERSION_2_56

causes QEMU to get compile time warnings (or errors) if it attempts to use a API feature deprecated in 2.56, even if the API exists in the header & library.

Quoted 6 lines

Note that it's not enough to replace "delete member" by "mark member deleted in <version>". You also have to keep full history for "is it optional". And for types, because those can evolve compatibly, too, e.g. from struct to flat union, or from string to alternate of string and something else. What is the proper version-spanning interface in all the possible cases?

Quoted 3 lines

I've not thought through all possible scenarios, but there may end up being restrictions, such that changes that were previously possible may have to be forbidden.

"There may be restrictions" is not exactly a confidence-inspring design assumption. We need a reasonably dependable idea on what exactly we're intending to sacrifice.

Quoted 19 lines

One example, in the past we could do deprecate a field 'foo', then delete 'foo' and then some time re-introduce 'foo' with a completely different type. That would not be possible if we wanted to maintain compat, but in this example that's probably a good thing, as it'll be super confusing to have the same field name change type like that over time. Easier to just use a different name.

So the question to me is not whether all our previous changes are still possible, but whether enough of the typwes of change are possible, such that we can cope with the ongoing maint in a reasonable way. I don't think we've explored the possibility enough to say one way or the other.

Quoted 5 lines
Quoted 3 lines

Apps that wish to have version compat, would of course need to write their code to be aware of which fields they need to seend for which QEMU version.

At which point we're reinventing libvirt.

Quoted 7 lines

The premise of the code generators is that there are applications that want to consume QEMU that are not libvirt. With this line of reasoning we could easily say that all such applications should just use libvirt and then we don't need to provide any of these code generators. The fact that we're considering these code generators though, says that we're accepting there are valid use cases that don't want to use libvirt for whatever reasons.

Can't resist any longer: "What has libvirt ever done for us?"
https://www.youtube.com/watch?v=Qc7HmhrgTuQ

Quoted 3 lines

It is
reasonable that some of those applications may wish to target a wide range of QEMU versions, just like libvirt does.

At which point they're comitting to reinventing the relevant parts of libvirt.

I'd expect marshalling and umarshalling QMP to be among the smaller sub-problems then. It may look relatively large at first, because it's among the first ones to be solved. More so when you hand-wave away the more interesting ones of *abstraction* until they bite you in the posterior.

Quoted 6 lines

It is also reasonable to say that libvirt would be better off if it could auto-generate a client API for QEMU too, instead of writing it by hand from a human reading the QAPI

With regards,
Daniel

#65 lore raw

Hi,

Quoted 27 lines · Markus Armbruster
Quoted 16 lines · Daniel P. Berrangé

Caller

    block_resize(device="dev0", size=1*GiB)
    block_resize(node_name="devnode0", size=1*GiB)

In golang definition

   type BlockResizeArguments struct {
       Device string
       NodeName string
       Size int
   }

Caller choice of

Quoted 9 lines
   cmd := &BlockResizeCommand{
       Device: "dev0",
       Size: 1 * GiB,
   }

   cmd := &BlockResizeCommand{
       NodeName: "devnode0",
       Size: 1 * GiB,
   }
Quoted 7 lines

Note that the Go bindings you sketched effectively use (poor man's) keyword arguments.

Quoted 2 lines

Neither case can easily prevent passing Device and NodeName at same time.

That defect lies at the schema's feet.

Right. The schema does not provide any metadata to explicit say that only @device or @node-name should be used, correct?

This would be important to differentiate of a simple 'adding a new optional argument' plus 'making this other argument optional'.

Quoted 49 lines
Quoted 14 lines
Quoted 2 lines

* At some future date, the old way gets deprecated: argument @device acquires feature @deprecated.

Ok, no change needed to the APIs in either case. Possibly have code emit a warning if a deprecated field is set.

Quoted 2 lines

* Still later, the old way gets removed: @device is deleted, and @node-name becomes mandatory.

Again no change needed to APIs, but QEMU will throw back an error if the wrong one is used.

Quoted 2 lines

What is the proper version-spanning interface?

Quoted 8 lines
Quoted 3 lines

I figure it's both arguments optional, must specify the right one for the version of QEMU actually in use. This spans versions, but it fails to abstract from them.

Yep, I think that's inevitable in this scenario. THe plus side is that apps that want to span versions can do so. The downside is that apps that don't want smarts to span version, may loose compile time warnings about use of the now deleted field.

The version-spanning interface will arguably be a bad interface for any version.

Quoted 23 lines

I suggested the code generator have an option to say what level of compat to use for generated code, so that apps can request an API without compat, which will result in compile errors. This though assumes every consumer app is embedding their own generated copy of the code. Not neccessarily desirable.

At the C level you can play games with __deprecated__ to get compile time warnings in some cases.

#define GLIB_VERSION_MIN_REQUIRED GLIB_VERSION_2_56

causes QEMU to get compile time warnings (or errors) if it attempts to use a API feature deprecated in 2.56, even if the API exists in the header & library.

Quoted 6 lines

Note that it's not enough to replace "delete member" by "mark member deleted in <version>". You also have to keep full history for "is it optional". And for types, because those can evolve compatibly, too, e.g. from struct to flat union, or from string to alternate of string and something else. What is the proper version-spanning interface in all the possible cases?

Quoted 7 lines
Quoted 3 lines

I've not thought through all possible scenarios, but there may end up being restrictions, such that changes that were previously possible may have to be forbidden.

"There may be restrictions" is not exactly a confidence-inspring design assumption. We need a reasonably dependable idea on what exactly we're intending to sacrifice.

I can't help much here but I guess we can evolve QAPI schema as we move forward. Adding metadata that helps document changes to the benefit of giving code generators tools to provide a way to work with those QAPI changes seems desirable, no?

Quoted 13 lines
Quoted 13 lines

One example, in the past we could do deprecate a field 'foo', then delete 'foo' and then some time re-introduce 'foo' with a completely different type. That would not be possible if we wanted to maintain compat, but in this example that's probably a good thing, as it'll be super confusing to have the same field name change type like that over time. Easier to just use a different name.

So the question to me is not whether all our previous changes are still possible, but whether enough of the typwes of change are possible, such that we can cope with the ongoing maint in a reasonable way. I don't think we've explored the possibility enough to say one way or the other.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

Apps that wish to have version compat, would of course need to write their code to be aware of which fields they need to seend for which QEMU version.

At which point we're reinventing libvirt.

IMHO, at this moment, qapi-go is targeting communicating with QEMU and handling multiple QEMU versions seems reasonable to me.

Perhaps libvirt can use qapi-go in the future or other generated interface. That would be cool.

Quoted 15 lines
Quoted 7 lines

The premise of the code generators is that there are applications that want to consume QEMU that are not libvirt. With this line of reasoning we could easily say that all such applications should just use libvirt and then we don't need to provide any of these code generators. The fact that we're considering these code generators though, says that we're accepting there are valid use cases that don't want to use libvirt for whatever reasons.

Can't resist any longer: "What has libvirt ever done for us?"
https://www.youtube.com/watch?v=Qc7HmhrgTuQ

Quoted 3 lines

It is
reasonable that some of those applications may wish to target a wide range of QEMU versions, just like libvirt does.

Quoted 8 lines

At which point they're comitting to reinventing the relevant parts of libvirt.

I'd expect marshalling and umarshalling QMP to be among the smaller sub-problems then. It may look relatively large at first, because it's among the first ones to be solved. More so when you hand-wave away the more interesting ones of *abstraction* until they bite you in the posterior.

I might have missed it but I don't see unsolvable problems.

1) We decide if we want a Golang interface that can communicate

    with multiple QEMU versions or not;

2) We discuss how that Golang interface would look like;
3) We discuss what is needed in QEMU/QAPI to achieve (2)
4) We work on QEMU/QAPI to address (3)
5) We move on with qapi-go proposal

I see only benefits with this project, plus the fact we already have Golang projects doing their own code to communicate with QEMU and I do believe we will make their lives easier.

Quoted 6 lines
Quoted 6 lines

It is also reasonable to say that libvirt would be better off if it could auto-generate a client API for QEMU too, instead of writing it by hand from a human reading the QAPI

With regards,
Daniel

Cheers,
Victor

#66 lore raw
Quoted 33 lines · Victor Toso

Hi,

Quoted 30 lines · Markus Armbruster
Quoted 16 lines · Daniel P. Berrangé

Caller

    block_resize(device="dev0", size=1*GiB)
    block_resize(node_name="devnode0", size=1*GiB)

In golang definition

   type BlockResizeArguments struct {
       Device string
       NodeName string
       Size int
   }

Caller choice of

Quoted 9 lines
   cmd := &BlockResizeCommand{
       Device: "dev0",
       Size: 1 * GiB,
   }

   cmd := &BlockResizeCommand{
       NodeName: "devnode0",
       Size: 1 * GiB,
   }

Note that the Go bindings you sketched effectively use (poor man's) keyword arguments.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Neither case can easily prevent passing Device and NodeName at same time.

That defect lies at the schema's feet.

Right. The schema does not provide any metadata to explicit say that only @device or @node-name should be used, correct?

Correct.

The existing means to express either / or are tagged unions and alternates.

Tagged unions require an explicit tag member.

Alternates select the alternative by the type of the value.

We don't have anything that selects by member name.

Quoted 3 lines

This would be important to differentiate of a simple 'adding a new optional argument' plus 'making this other argument optional'.

We also don't have means to express "this integer must be a power of two", or "this string must name a block backend", or any number of semantic constraints.

We have to draw the line somewhere. Schema language complexity needs to earn its keep.

Quoted 53 lines
Quoted 53 lines
Quoted 14 lines
Quoted 2 lines

* At some future date, the old way gets deprecated: argument @device acquires feature @deprecated.

Ok, no change needed to the APIs in either case. Possibly have code emit a warning if a deprecated field is set.

Quoted 2 lines

* Still later, the old way gets removed: @device is deleted, and @node-name becomes mandatory.

Again no change needed to APIs, but QEMU will throw back an error if the wrong one is used.

Quoted 2 lines

What is the proper version-spanning interface?

Quoted 8 lines
Quoted 3 lines

I figure it's both arguments optional, must specify the right one for the version of QEMU actually in use. This spans versions, but it fails to abstract from them.

Yep, I think that's inevitable in this scenario. THe plus side is that apps that want to span versions can do so. The downside is that apps that don't want smarts to span version, may loose compile time warnings about use of the now deleted field.

The version-spanning interface will arguably be a bad interface for any version.

Quoted 26 lines

I suggested the code generator have an option to say what level of compat to use for generated code, so that apps can request an API without compat, which will result in compile errors. This though assumes every consumer app is embedding their own generated copy of the code. Not neccessarily desirable.

At the C level you can play games with __deprecated__ to get compile time warnings in some cases.

#define GLIB_VERSION_MIN_REQUIRED GLIB_VERSION_2_56

causes QEMU to get compile time warnings (or errors) if it attempts to use a API feature deprecated in 2.56, even if the API exists in the header & library.

Quoted 6 lines

Note that it's not enough to replace "delete member" by "mark member deleted in <version>". You also have to keep full history for "is it optional". And for types, because those can evolve compatibly, too, e.g. from struct to flat union, or from string to alternate of string and something else. What is the proper version-spanning interface in all the possible cases?

I've not thought through all possible scenarios, but there may end up being restrictions, such that changes that were previously possible may have to be forbidden.

Quoted 8 lines
Quoted 3 lines

"There may be restrictions" is not exactly a confidence-inspring design assumption. We need a reasonably dependable idea on what exactly we're intending to sacrifice.

I can't help much here but I guess we can evolve QAPI schema as we move forward. Adding metadata that helps document changes to the benefit of giving code generators tools to provide a way to work with those QAPI changes seems desirable, no?

QMP comes with a certain compatibility promise. Keeping the promise has been expensive, but that's okay; a weaker promise would have been differently expensive. It's a compromise that has emerged over a long time.

I fail to see why QEMU-provided QMP bindings should come with a stronger promise than QEMU-provided QMP.

Quoted 13 lines
Quoted 13 lines
Quoted 13 lines

One example, in the past we could do deprecate a field 'foo', then delete 'foo' and then some time re-introduce 'foo' with a completely different type. That would not be possible if we wanted to maintain compat, but in this example that's probably a good thing, as it'll be super confusing to have the same field name change type like that over time. Easier to just use a different name.

So the question to me is not whether all our previous changes are still possible, but whether enough of the typwes of change are possible, such that we can cope with the ongoing maint in a reasonable way. I don't think we've explored the possibility enough to say one way or the other.

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

Apps that wish to have version compat, would of course need to write their code to be aware of which fields they need to seend for which QEMU version.

At which point we're reinventing libvirt.

IMHO, at this moment, qapi-go is targeting communicating with QEMU and handling multiple QEMU versions seems reasonable to me.

It's targeting communicating in *QMP*. QMP is designed to support communicating with a range of QEMU versions. Full compatibility is promised for a narrow range. Outside that range, graceful degradation.

*If* you want to widen the full compatibility range, do it in *QMP*. Or do it on top of QEMU, e.g. in libvirt.

Quoted 2 lines

Perhaps libvirt can use qapi-go in the future or other generated interface. That would be cool.

"Would be cool" and a dollar buys you a cup of bad coffee.

Is it a good use of our limited resources?

How much will it delay delivery of Go bindings compared to less ambitious version?

Quoted 24 lines
Quoted 15 lines
Quoted 7 lines

The premise of the code generators is that there are applications that want to consume QEMU that are not libvirt. With this line of reasoning we could easily say that all such applications should just use libvirt and then we don't need to provide any of these code generators. The fact that we're considering these code generators though, says that we're accepting there are valid use cases that don't want to use libvirt for whatever reasons.

Can't resist any longer: "What has libvirt ever done for us?"
https://www.youtube.com/watch?v=Qc7HmhrgTuQ

Quoted 3 lines

It is
reasonable that some of those applications may wish to target a wide range of QEMU versions, just like libvirt does.

Quoted 8 lines

At which point they're comitting to reinventing the relevant parts of libvirt.

I'd expect marshalling and umarshalling QMP to be among the smaller sub-problems then. It may look relatively large at first, because it's among the first ones to be solved. More so when you hand-wave away the more interesting ones of *abstraction* until they bite you in the posterior.

Quoted 8 lines

I might have missed it but I don't see unsolvable problems.

1) We decide if we want a Golang interface that can communicate

    with multiple QEMU versions or not;

2) We discuss how that Golang interface would look like;
3) We discuss what is needed in QEMU/QAPI to achieve (2)
4) We work on QEMU/QAPI to address (3)
5) We move on with qapi-go proposal

This reinvents significant parts of libvirt in QEMU.

Quoted 3 lines

I see only benefits with this project, plus the fact we already have Golang projects doing their own code to communicate with QEMU and I do believe we will make their lives easier.

Perhaps my idea of practical Go usage is hopelessly naive, but here goes anyway.

Go project vendors current Go bindings (because they vendor everything). Works with current QEMU, plus a range of past and future QEMUs.

Eventually something breaks with a then current QEMU. Go project updates vendored Go bindings to current. Compiler points out everything they need fixing (this assumes the type system is sufficiently expressive, and bindings use it properly).

Good enough for these Golang projects?

Or asked in a different way: what exactly are the compatibility requirements of these projects?

Please keep it to requirements *actual* projects have. Providing for additional, non-trivial requirements hypothetical projects might have is, in my reasoned opinion, folly we cannot afford. YAGNI until demonstrated otherwise.

If different projects have different requirements, then QEMU having to provide for all of them does not follow. Nothing absolves us from weighing benefits vs. costs and risks.

#67 lore raw
Quoted 14 lines · Markus Armbruster
Quoted 2 lines · Victor Toso

IMHO, at this moment, qapi-go is targeting communicating with QEMU and handling multiple QEMU versions seems reasonable to me.

It's targeting communicating in *QMP*. QMP is designed to support communicating with a range of QEMU versions. Full compatibility is promised for a narrow range. Outside that range, graceful degradation.

*If* you want to widen the full compatibility range, do it in *QMP*. Or do it on top of QEMU, e.g. in libvirt.

Quoted 2 lines

Perhaps libvirt can use qapi-go in the future or other generated interface. That would be cool.

Quoted 6 lines

"Would be cool" and a dollar buys you a cup of bad coffee.

Is it a good use of our limited resources?

How much will it delay delivery of Go bindings compared to less ambitious version?

Yeah, this thread has basically branched to cover three topics:

1. what an MVP Go interface for QMP should look like;
2. how to make sure said interface uses pretty names;
3. how to make it work across multiple QEMU versions.

All of these are important in the long run, but as far as I'm concerned only 1. is an actual blocker to making progress.

If we get to the point where we can generate a reasonably complete and well-typed Go interface that can be used to communicate with a single version of QEMU, we should just plaster EXPERIMENTAL all over it and get it merged.

Basically get the MVP done and then iterate over it in-tree rather than trying to get everything perfect from the start.

Sounds reasonable?

Signature · 1 line
Andrea Bolognani / Red Hat / Virtualization
#68 lore raw
Quoted 36 lines · Andrea Bolognani
Quoted 14 lines · Markus Armbruster
Quoted 2 lines · Victor Toso

IMHO, at this moment, qapi-go is targeting communicating with QEMU and handling multiple QEMU versions seems reasonable to me.

It's targeting communicating in *QMP*. QMP is designed to support communicating with a range of QEMU versions. Full compatibility is promised for a narrow range. Outside that range, graceful degradation.

*If* you want to widen the full compatibility range, do it in *QMP*. Or do it on top of QEMU, e.g. in libvirt.

Quoted 2 lines

Perhaps libvirt can use qapi-go in the future or other generated interface. That would be cool.

Quoted 6 lines

"Would be cool" and a dollar buys you a cup of bad coffee.

Is it a good use of our limited resources?

How much will it delay delivery of Go bindings compared to less ambitious version?

Yeah, this thread has basically branched to cover three topics:

1. what an MVP Go interface for QMP should look like;
2. how to make sure said interface uses pretty names;
3. how to make it work across multiple QEMU versions.

All of these are important in the long run, but as far as I'm concerned only 1. is an actual blocker to making progress.

If we get to the point where we can generate a reasonably complete and well-typed Go interface that can be used to communicate with a single version of QEMU, we should just plaster EXPERIMENTAL all over it and get it merged.

Quoted 4 lines

Basically get the MVP done and then iterate over it in-tree rather than trying to get everything perfect from the start.

Sounds reasonable?

Yes, with an undogmatic interpretation of "minimally viable". Doing more should be okay when it doesn't complicate things outside the Go-generating backend.

Exploring how to generate Go bindings that make good use of static typing will be interesting enough. Aiming for wide compatibility in addition risks delay and/or failure. Exploding heads, too.

#69 lore raw
In reply to Andrea Bolognani

Hi,

Quoted 22 lines · Andrea Bolognani
Quoted 14 lines · Markus Armbruster
Quoted 2 lines · Victor Toso

IMHO, at this moment, qapi-go is targeting communicating with QEMU and handling multiple QEMU versions seems reasonable to me.

It's targeting communicating in *QMP*. QMP is designed to support communicating with a range of QEMU versions. Full compatibility is promised for a narrow range. Outside that range, graceful degradation.

*If* you want to widen the full compatibility range, do it in *QMP*. Or do it on top of QEMU, e.g. in libvirt.

Quoted 2 lines

Perhaps libvirt can use qapi-go in the future or other generated interface. That would be cool.

Quoted 6 lines

"Would be cool" and a dollar buys you a cup of bad coffee.

Is it a good use of our limited resources?

How much will it delay delivery of Go bindings compared to less ambitious version?

Quoted 8 lines

Yeah, this thread has basically branched to cover three topics:

1. what an MVP Go interface for QMP should look like;
2. how to make sure said interface uses pretty names;
3. how to make it work across multiple QEMU versions.

All of these are important in the long run, but as far as I'm concerned only 1. is an actual blocker to making progress.

I agree although (1) and (3) are holding hands a bit.

Quoted 9 lines

If we get to the point where we can generate a reasonably complete and well-typed Go interface that can be used to communicate with a single version of QEMU, we should just plaster EXPERIMENTAL all over it and get it merged.

Basically get the MVP done and then iterate over it in-tree rather than trying to get everything perfect from the start.

Sounds reasonable?

Yep. The whole discussion has been great as to clarify limitations and possible goals but not aiming to get it perfect all at once seems reasonable.

Cheers,
Victor

#70 lore raw
In reply to Victor Toso
Quoted 9 lines · Victor Toso

* Status

There are a few rough edges to work on but this is usable. The major thing I forgot to add is handling Error from Commands. It'll be the first thing I'll work on next week.

If you want to start using this Today you can fetch it in at

https://gitlab.com/victortoso/qapi-go/

Looking at this my main concern is that there is way too much use of the 'Any' type. The valid Golang types that can be used with any of the 'Any' fields are merely expresssed as comments in the code. I think this needs changing so that the Golang types are directly expressed in code.

I think there are perhaps only 2-3 cases where the 'Any' type is genuinely neccessary due to the QAPI schema having an unbounded set of possible types - SchemaInfoObjectMember, ObjectPropertyInfo and QomSetCommand

With regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|