Skip to contentSkip to conversation

Proposal: how to make incompatible changes

DISCUSSION Joseph Reynolds 6 participants 11 messages Apr 14, 2020 – May 4, 2020

lore
Share & tools
lore
#0 lore raw

Proposal: how to make incompatible changes

This is a proposal to add guidelines for making incompatible changes to OpenBMC interfaces.  Is it okay to make incompatible changes? Yes, IMHO: the project will continuously break compatibility in various ways, and its users will adapt.  The main idea is to minimize churn and make it easier for users to adapt.

As the OpenBMC project moves forward with new releases, it will make changes that necessarily break existing use cases.  My recommendations are:
- Try hard to maintain forward compatibility.  For example, maintain all of the BMC's intended user interfaces.
- Identify changes that break compatibility.  Briefly describe the use case, what breaks, how a user can adapt, and cross-link technical discussions (Gerrit reviews, issues, emails).
- Work with maintainers to determine which incompatible changes get merged and what documentation is needed.
- Give users time to adapt to incompatible changes.  For example, deprecate interfaces in a previous release.
- List incompatible changes in the [release notes][] so community members will know they have to adapt, and link to how to adapt.

[release notes]: https://github.com/openbmc/docs/blob/master/release/release-notes

Perhaps we could add a section to one of these here documents:
https://github.com/openbmc/docs/blob/master/maintainer-workflow.md
https://github.com/openbmc/docs/blob/master/CONTRIBUTING.md

What do you think?

- Joseph

TLDR:

Applicability.

These guidelines are for the BMC's "intended external user interfaces".  For example, its management interfaces including its web server and all REST APIs.  I haven't given much thought to the BMC/host interfaces or interfaces internal to the BMC.  IMHO, it is less important to maintain compatibility in these areas.  For example, if you need an incompatible change in an internal interface, you have a smaller set of users who ought to be active in the project, and can give you feedback and adapt within a release cycle.

What is forward compatibility?

The OpenBMC project has a sequence of releases.  When someone is using one of these releases, then updates to a later release, ideally all the BMC interfaces they use will continue to work as before.  That's what I call "forward compatibility"  which is close to the Wikipedia's [upward compatibility][].

Backward compatibility means that after you perform a firmware update to an older release in the sequence, using that release's older interfaces will continue to operate the BMC as before.  That is, you can downgrade to that release.  For example, backward compatibility is needed to be able to revert a previous update back to a known-good firmware image version.  Once again, this is different from Wikipedia's [backward compatibility][] entry which is more like interoperability.

I think it is useful to separate these concepts.  To be clear: I am in favor of the OpenBMC interfaces being upward compatible, backward compatible, and interoperable, but I think we need to take baby steps to get there.  So I am limiting the scope of this email to a narrow definition of forward compatibility.

I expect maintainers will consider backward compatibility at the same time they consider forward compatibility.  Note that it may be acceptable to break backward compatibility more frequently that breaking forward compatibility.  For example, you can tell users that once they upgrade to a new release, they cannot downgrade.

[upward compatibility]: https://en.wikipedia.org/wiki/Forward_compatibility
[backward compatibility]: https://en.wikipedia.org/wiki/Backward_compatibility

Examples of changes that may break compatibility:

https://gerrit.openbmc-project.xyz/c/openbmc/pam-ipmi/+/31054

https://github.com/openbmc/openbmc/issues/3615

https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/29344

#1 lore raw
Quoted 7 lines

Applicability.

These guidelines are for the BMC's "intended external user interfaces". For example, its management interfaces including its web server and all REST APIs.  I haven't given much thought to the BMC/host interfaces or interfaces internal to the BMC.  IMHO, it is less important to maintain compatibility in these areas.

Lets split this. My feelings are

1. Inband (BMC/Host) interfaces are in the same class as "intended external user interfaces" and therefore should not have incompatible changes unless _absolutely_ necessary. What we implement here should have passed through a specification process under e.g. DMTF.

2. Interfaces between applications on the BMC (e.g. D-Bus interfaces) is the class where compatibility is less critical, on systems that do not expose the D-Bus interfaces via REST. However, while any system exists that exposes the D-Bus interfaces via REST we must constrain changes to these interfaces as well.

Quoted 5 lines

For example, if you need an incompatible
change in an internal interface, you have a smaller set of users who ought to be active in the project, and can give you feedback and adapt within a release cycle.

This is true for interfaces between BMC components, it's not true of the host firmware interfaces, hence my split above.

When we remove the ability to directly access the D-Bus interfaces via REST we will gain a lot more freedom as the D-Bus interfaces then truly become internal.

Andrew

#2 lore raw

Thank you for bringing this up. I really think that some policy discussion is useful here. I also 100% agree that forward compatibility should be our goal here.

When I think about incompatible changes I tend to group them into one of two categories:
1) A design change goes in the opposite direction than what previous versions offered, and the old feature needs to be shut off.
2) New features/designs are made, and supporting several different options at the same time is difficult

Security changes are often in the former category. I remember a few months back there was a thread about removing the DBus rest interface as a default service for security reasons. Brad's point at the time was (I may be paraphrasing here) that we need to maintain support for any currently supported use case or be able to support their migration.

Now another point in that thread was that some users are going to care more about security than compatibility, and vice versa. One possible solution is to create a second secure phosphor reference implementation.

Here is the thread I'm talking about:
https://lists.ozlabs.org/pipermail/openbmc/2020-February/020491.html

Having two reference implementations (profiles, tracks, or whatever we want to call it) isn't without risk. If things ever diverge too far, then the overhead might be larger than supporting forward compatibility. However, I would like to get to a place where changes of that sort can be made accessible to early adopters, combined with a clearly communicated deprecation/migration plan.

Quoted 36 lines · Andrew Jeffery
Quoted 7 lines

Applicability.

These guidelines are for the BMC's "intended external user interfaces". For example, its management interfaces including its web server and all REST APIs. I haven't given much thought to the BMC/host interfaces or interfaces internal to the BMC. IMHO, it is less important to maintain compatibility in these areas.

Lets split this. My feelings are

1. Inband (BMC/Host) interfaces are in the same class as "intended external user interfaces" and therefore should not have incompatible changes unless _absolutely_ necessary. What we implement here should have passed through a specification process under e.g. DMTF.

2. Interfaces between applications on the BMC (e.g. D-Bus interfaces) is the class where compatibility is less critical, on systems that do not expose the D-Bus interfaces via REST. However, while any system exists that exposes the D-Bus interfaces via REST we must constrain changes to these interfaces as well.

Quoted 5 lines

For example, if you need an incompatible
change in an internal interface, you have a smaller set of users who ought to be active in the project, and can give you feedback and adapt within a release cycle.

This is true for interfaces between BMC components, it's not true of the host firmware interfaces, hence my split above.

When we remove the ability to directly access the D-Bus interfaces via REST we will gain a lot more freedom as the D-Bus interfaces then truly become internal.

Andrew

#3 lore raw

On 4/16/20 4:53 PM, Richard Hanley wrote:  -- also responding to Andrew Jeffery

Quoted 7 lines

Thank you for bringing this up.  I really think that some policy discussion is useful here.  I also 100% agree that forward compatibility should be our goal here.

When I think about incompatible changes I tend to group them into one of two categories:
1) A design change goes in the opposite direction than what previous versions offered, and the old feature needs to be shut off.
2) New features/designs are made, and supporting several different options at the same time is difficult

Security changes are often in the former category. I remember a few months back there was a thread about removing the DBus rest interface as a default service for security reasons.  Brad's point at the time was (I may be paraphrasing here) that we need to maintain support for any currently supported use case or be able to support their migration.

There is broad support to disable the phosphor REST APIs by default.  I understand we are waiting for a little more development work so this will go smoothly.  See comments in the review:
https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/29344

Quoted 1 line

Now another point in that thread was that some users are going to care more about security than compatibility, and vice versa. One possible solution is to create a second secure phosphor reference implementation.

I am trying hard to move the OpenBMC distro to be "secure by default".  If we run into compatibility issues, I would try to put them in as DISTRO_FEATUREs, and then work to change the default to the secure setting.

An example for this is network IPMI.  I am (we are?) working toward allowing the BMC admin to disable the network IPMI service, but it would be enabled by default.  Then in future, we can change the default to disabled. ... (goes to look for old email)....

So I don't see a need for a second reference distro.

Quoted 20 lines

Here is the thread I'm talking about: https://lists.ozlabs.org/pipermail/openbmc/2020-February/020491.html

Having two reference implementations (profiles, tracks, or whatever we want to call it) isn't without risk.  If things ever diverge too far, then the overhead might be larger than supporting forward compatibility.  However, I would like to get to a place where changes of that sort can be made accessible to early adopters, combined with a clearly communicated deprecation/migration plan.

Quoted 3 lines · On Wed, Apr 15, 2020 at 10:01 PM Andrew Jeffery <[email protected]…

Applicability.

These guidelines are for the BMC's "intended external user

interfaces".
> For example, its management interfaces including its web server and all
> REST APIs.  I haven't given much thought to the BMC/host interfaces or
> interfaces internal to the BMC.  IMHO, it is less important to maintain
> compatibility in these areas.

Quoted 7 lines

Lets split this. My feelings are

1. Inband (BMC/Host) interfaces are in the same class as "intended external user interfaces" and therefore should not have incompatible changes unless _absolutely_ necessary. What we implement here should have passed through a specification process under e.g. DMTF.

That sounds right to me.  I have less understanding of the BMC/host interface, so I'm ahpy ot let someone else lead there.

Quoted 8 lines

2. Interfaces between applications on the BMC (e.g. D-Bus interfaces) is the class where compatibility is less critical, on systems that do not
expose the D-Bus interfaces via REST. However, while any system exists that exposes the D-Bus interfaces via REST we must constrain changes to these interfaces as well.

Agreed.

Quoted 20 lines
Quoted 3 lines

For example, if you need an incompatible
change in an internal interface, you have a smaller set of users

who
> ought to be active in the project, and can give you feedback and adapt
> within a release cycle.
>

This is true for interfaces between BMC components, it's not true of the
host firmware interfaces, hence my split above.

When we remove the ability to directly access the D-Bus interfaces via REST we will gain a lot more freedom as the D-Bus interfaces then truly
become internal.

Andrew

#4 lore raw
In reply to Joseph Reynolds

15.04.2020 02:00, Joseph Reynolds пишет:

Quoted 4 lines

Proposal: how to make incompatible changes

This is a proposal to add guidelines for making incompatible changes to OpenBMC interfaces.  Is it okay to make incompatible changes? Yes, IMHO: the project will continuously break compatibility in various ways, and its users will adapt.  The main idea is to minimize churn and make it easier for users to adapt.

Quoted 13 lines

As the OpenBMC project moves forward with new releases, it will make changes that necessarily break existing use cases.  My recommendations are:
- Try hard to maintain forward compatibility.  For example, maintain
all of the BMC's intended user interfaces.
- Identify changes that break compatibility.  Briefly describe the use
case, what breaks, how a user can adapt, and cross-link technical discussions (Gerrit reviews, issues, emails).
- Work with maintainers to determine which incompatible changes get
merged and what documentation is needed.
- Give users time to adapt to incompatible changes.  For example,
deprecate interfaces in a previous release.
- List incompatible changes in the [release notes][] so community
members will know they have to adapt, and link to how to adapt.

I'd say that any incompatible change must be reflected in the interface version.

As far as I understand, RedFish has all the interfaces strictly specified and those specifications are bound to a specific RedFish version that is then encoded in the URL (e.g. /redfish/v1/).

Any other interface should have a similar approach. A more relaxed one, if we keep our own flavor of REST, could be that each interface could have a Version property (probably also encoded in the REST URL).

That way, any user trying to access an old interface (thus via a non-existant URL) would get a 404.

Internally, for inter-process dbus communication the interface version could be checked during compile time to prevent problems that couldn't be detected by compiler/linker automatically. Those problems are, for instance, incompatible change in the meaning of the values of a property.

Alexander.

#5 lore raw

Re: Proposal: how to make incompatible changes - versioning

Quoted 20 lines · Alexander Amelkin

15.04.2020 02:00, Joseph Reynolds пишет:

Quoted 4 lines

Proposal: how to make incompatible changes

This is a proposal to add guidelines for making incompatible changes to OpenBMC interfaces.  Is it okay to make incompatible changes? Yes, IMHO: the project will continuously break compatibility in various ways, and its users will adapt.  The main idea is to minimize churn and make it easier for users to adapt.

Quoted 13 lines

As the OpenBMC project moves forward with new releases, it will make changes that necessarily break existing use cases.  My recommendations are:
- Try hard to maintain forward compatibility.  For example, maintain
all of the BMC's intended user interfaces.
- Identify changes that break compatibility.  Briefly describe the
use case, what breaks, how a user can adapt, and cross-link technical discussions (Gerrit reviews, issues, emails).
- Work with maintainers to determine which incompatible changes get
merged and what documentation is needed.
- Give users time to adapt to incompatible changes.  For example,
deprecate interfaces in a previous release.
- List incompatible changes in the [release notes][] so community
members will know they have to adapt, and link to how to adapt.

I'd say that any incompatible change must be reflected in the interface version.

Quoted 7 lines

As far as I understand, RedFish has all the interfaces strictly specified and those specifications are bound to a specific RedFish version that is then encoded in the URL (e.g. /redfish/v1/).

Any other interface should have a similar approach. A more relaxed one, if we keep our own flavor of REST, could be that each interface could have a Version property (probably also encoded in the REST URL).

That way, any user trying to access an old interface (thus via a non-existant URL) would get a 404.

Internally, for inter-process dbus communication the interface version could be checked during compile time to prevent problems that couldn't be detected by compiler/linker automatically. Those problems are, for instance, incompatible change in the meaning of the values of a property.

Agreed.  The question to ponder: What is our REST API versioning strategy?  What  best practices do we follow?
- Joseph

Quoted 2 lines

Alexander.

#6 lore raw

Re: Proposal: how to make incompatible changes - versioning

Quoted 40 lines · Joseph Reynolds
Quoted 39 lines · Alexander Amelkin

15.04.2020 02:00, Joseph Reynolds пишет:

Quoted 8 lines

Proposal: how to make incompatible changes

This is a proposal to add guidelines for making incompatible changes to OpenBMC interfaces.  Is it okay to make incompatible changes? Yes,
IMHO: the project will continuously break compatibility in various ways, and its users will adapt.  The main idea is to minimize churn and make it easier for users to adapt.

Quoted 16 lines

As the OpenBMC project moves forward with new releases, it will make changes that necessarily break existing use cases.  My recommendations are:
- Try hard to maintain forward compatibility.  For example, maintain all of the BMC's intended user interfaces.
- Identify changes that break compatibility.  Briefly describe the use case, what breaks, how a user can adapt, and cross-link technical discussions (Gerrit reviews, issues, emails).
- Work with maintainers to determine which incompatible changes get merged and what documentation is needed.
- Give users time to adapt to incompatible changes.  For example, deprecate interfaces in a previous release.
- List incompatible changes in the [release notes][] so community members will know they have to adapt, and link to how to adapt.

I'd say that any incompatible change must be reflected in the interface version.

As far as I understand, RedFish has all the interfaces strictly specified and those specifications are bound to a specific RedFish version that is then encoded in the URL (e.g. /redfish/v1/).

Any other interface should have a similar approach. A more relaxed one, if we keep our own flavor of REST, could be that each interface could have a Version property (probably also encoded in the REST URL).

That way, any user trying to access an old interface (thus via a non-existant URL) would get a 404.

Quoted 7 lines
Quoted 4 lines

Internally, for inter-process dbus communication the interface version could be checked during compile time to prevent problems that couldn't be detected by compiler/linker automatically. Those problems are, for instance, incompatible change in the meaning of the values of a property.

Agreed.  The question to ponder: What is our REST API versioning strategy?  What  best practices do we follow?

The D-Bus specification already recommends how to version D-Bus interfaces:

https://dbus.freedesktop.org/doc/dbus-specification.html#message-protocol-names-interface

Andrew

#7 lore raw
In reply to Alexander Amelkin
Quoted 1 line · Alexander Amelkin

Internally, for inter-process dbus communication the interface version could be checked during compile time to prevent problems that couldn't be detected by compiler/linker automatically.

I like the idea of a stable, versioned DBus API. Does anyone else?

In the past there was opposition to that. I’m not sure if there still is.

-brad

#8 lore raw
Quoted 9 lines · Brad Bishop
Quoted 3 lines · Alexander Amelkin

Internally, for inter-process dbus communication the interface version could be checked during compile time to prevent problems that couldn't be detected by compiler/linker automatically.

I like the idea of a stable, versioned DBus API. Does anyone else?

In the past there was opposition to that. I’m not sure if there still is.

Since we're deprecating the REST-dbus path and moving towards Redfish is there much advantage to a versioned DBus API? Alexander mentioned compile-time checking of the interface, but we already have that through sdbusplus. The issues are:
1. Client bindings are not currently being generated.
2. Not every implementation is using them.
3. We don't have a good mechanism to deal with cross-repository interface changes.

I don't think a versioned DBus API solves any of these, except maybe #3 but that is only if we're going to write servers that support N-versions of the interfaces.

It seems to me like a better investment is #1 and #2? I'd personally like to have #1 implemented by the end of Sept at the latest. Solving #3, I think, requires us to do some CI investment to support Gerrit topic-based testing.

Signature · 1 line
Patrick Williams
#9 lore raw
Quoted 7 lines · Patrick Williams
Quoted 6 lines · Brad Bishop
Quoted 3 lines · Alexander Amelkin

Internally, for inter-process dbus communication the interface version could be checked during compile time to prevent problems that couldn't be detected by compiler/linker automatically.

Quoted 6 lines
Quoted 3 lines

I like the idea of a stable, versioned DBus API. Does anyone else?

In the past there was opposition to that. I’m not sure if there still is.

Since we're deprecating the REST-dbus path and moving towards Redfish is there much advantage to a versioned DBus API?

It helps lower the barrier to comprehension and entry. Unversioned & unstable IPC APIs are not normal and probably shocking to anyone that comes across them.

An unstable, unversioned API encourages bad behavior - e.g. I can just make this in-compatible change to the API and fix all the applications at the same time. lock-step dependencies between 3 different projects (phosphor-dbus-interfaces, the client, and the server) is not a good thing.

Put another way - with an unstable and unversioned API - what is the point of having all these different projects and repositories? It just becomes one giant monolithic code base. If that is what we want, then we should structure it that way. Nothing inherently wrong with that, but it seems like it would restrict how the code can be used. Does that bother us?

Quoted 9 lines

Alexander mentioned
compile-time checking of the interface, but we already have that through sdbusplus. The issues are:
1. Client bindings are not currently being generated.
2. Not every implementation is using them.
3. We don't have a good mechanism to deal with cross-repository interface changes.

I don't think a versioned DBus API solves any of these, except maybe #3

That's just it - these are the solution to the problems that an unstable and unversioned API bring, not the other way around. It is all work one way or the other - we can do something atypical like have an unversioned and unstable API, and then write a bunch of frameworks and tooling around that so that it can work. Or we can put in the effort to have a stable and versioned DBus API. I would rather do the latter because that is how people have been using dbus effectively for a really long time.

Quoted 4 lines

but that is only if we're going to write servers that support N-versions of the interfaces.

It seems to me like a better investment is #1 and #2?

To be clear on #2, use of frameworks is an optional thing. To get universal adoption of a framework, you have to first get universal acceptance of the merits of the framework. So I suggest interlocking with the specific application maintainers before working on #2 to avoid wasting time.

Quoted 2 lines

I'd personally
like to have #1 implemented by the end of Sept at the latest.

Nice!

Quoted 3 lines

Solving
#3, I think, requires us to do some CI investment to support Gerrit topic-based testing.

Or we can avoid entirely, with a stable, versioned API :-)

#10 lore raw
Quoted 63 lines · Brad Bishop
Quoted 7 lines · Patrick Williams
Quoted 6 lines · Brad Bishop
Quoted 3 lines · Alexander Amelkin

Internally, for inter-process dbus communication the interface version could be checked during compile time to prevent problems that couldn't be detected by compiler/linker automatically.

Quoted 6 lines
Quoted 3 lines

I like the idea of a stable, versioned DBus API. Does anyone else?

In the past there was opposition to that. I’m not sure if there still is.

Since we're deprecating the REST-dbus path and moving towards Redfish is there much advantage to a versioned DBus API?

It helps lower the barrier to comprehension and entry. Unversioned & unstable IPC APIs are not normal and probably shocking to anyone that comes across them.

An unstable, unversioned API encourages bad behavior - e.g. I can just make this in-compatible change to the API and fix all the applications at the same time. lock-step dependencies between 3 different projects (phosphor-dbus-interfaces, the client, and the server) is not a good thing.

Put another way - with an unstable and unversioned API - what is the point of having all these different projects and repositories? It just becomes one giant monolithic code base. If that is what we want, then we should structure it that way. Nothing inherently wrong with that, but it seems like it would restrict how the code can be used. Does that bother us?

Quoted 9 lines

Alexander mentioned
compile-time checking of the interface, but we already have that through sdbusplus. The issues are:
1. Client bindings are not currently being generated.
2. Not every implementation is using them.
3. We don't have a good mechanism to deal with cross-repository interface changes.

I don't think a versioned DBus API solves any of these, except maybe #3

That's just it - these are the solution to the problems that an unstable and unversioned API bring, not the other way around. It is all work one way or the other - we can do something atypical like have an unversioned and unstable API, and then write a bunch of frameworks and tooling around that so that it can work. Or we can put in the effort to have a stable and versioned DBus API. I would rather do the latter because that is how people have been using dbus effectively for a really long time.

Quoted 4 lines

but that is only if we're going to write servers that support N-versions of the interfaces.

It seems to me like a better investment is #1 and #2?

To be clear on #2, use of frameworks is an optional thing. To get universal adoption of a framework, you have to first get universal acceptance of the merits of the framework. So I suggest interlocking with the specific application maintainers before working on #2 to avoid wasting time.

Quoted 2 lines

I'd personally
like to have #1 implemented by the end of Sept at the latest.

Quoted 8 lines

Nice!

Quoted 3 lines

Solving
#3, I think, requires us to do some CI investment to support Gerrit topic-based testing.

Or we can avoid entirely, with a stable, versioned API :-)

+1 for stable, versioned APIs.