Skip to contentSkip to conversation

Can we improve the experience of working on OpenBMC?

DISCUSSION Andrew Jeffery 5 participants 10 messages Jul 27, 2022 – Aug 9, 2022

lore
Share & tools
lore
#0 lore raw

Hello everyone,

A few months back Ed kicked off a thread about changing how we work on OpenBMC with the aim to improve the development rate and make it easier to on-board people to the project:

https://lore.kernel.org/openbmc/CAH2-KxAJS_U8=meCxp8ue7n0bmnzeRpyZOPZpy0h1cFEbbz-HA@mail.gmail.com/

I felt that discussion splintered a bit, with a lot of back-and-forth about details well down in the weeds. I found this hard to follow, and so put in some work to try synthesise the discussion into desires for improving how we work, and practical problems with what we do currently.

Below are the lists of these desires and problems. I think it would be good to treat this as a survey to understand who feels strongly about what.

If you want to express support for a point then add a +1 below it. Feel free to suggest new items for either list, or to further discuss a particular item below its entry.

I feel that with information on what people feel strongly about we can prioritise what we need address and work forwards from there.

For the record, the mind map that I used to generate these lists is here, which contains further quotes and references to the original discussion thread:

https://github.com/amboar/openbmc-monorepo-discussion

Some of these might be closely related or considered duplicates of other list items, but based on the discussions referenced above I felt they were distinct enough to warrant separate entries.

# Desires

1. Easy sharing of a broad set of application and distro changes
2. Minimise reviews required to get a given feature or fix integrated into the distro build
3. Make fork maintenance easy
4. Provide one place to report bugs across the entire project
5. Minimise effort collecting project statistics
6. Make it easy to add new applications
7. Make it easy to refactor across the entire project
8. Support inclusive naming

# Problems

1. Yocto is hard
1. Managing patch stacks is hard
2. Yocto-specific tooling is hard
3. Finding the right recipe file to inspect/modify is hard
4. Yocto has too much documentation
2. OpenBMC has too much documentation
3. Querying design/implementation/bug properties across the project is hard
4. Coordinating breaking changes is hard
5. Coordinating tree-wide changes is hard
6. Identifying the right repo to file a bug against is hard
7. Transferring bugs between repos is hard
8. Bug reports are duplicated across repos
9. Bug reports are ignored
10. Working out where to submit a patch is hard
11. Getting patches reviewed is hard
12. New repo requests are bottle-necked

Cheers,

Andrew

#1 lore raw

Hi Andrew, thanks for poking at this.

Quoted 28 lines · Andrew Jeffery

Hello everyone,

A few months back Ed kicked off a thread about changing how we work on OpenBMC with the aim to improve the development rate and make it easier to on-board people to the project:

https://lore.kernel.org/openbmc/CAH2-KxAJS_U8=meCxp8ue7n0bmnzeRpyZOPZpy0h1cFEbbz-HA@mail.gmail.com/

I felt that discussion splintered a bit, with a lot of back-and-forth about details well down in the weeds. I found this hard to follow, and so put in some work to try synthesise the discussion into desires for improving how we work, and practical problems with what we do currently.

Below are the lists of these desires and problems. I think it would be good to treat this as a survey to understand who feels strongly about what.

If you want to express support for a point then add a +1 below it. Feel free to suggest new items for either list, or to further discuss a particular item below its entry.

I feel that with information on what people feel strongly about we can prioritise what we need address and work forwards from there.

For the record, the mind map that I used to generate these lists is here, which contains further quotes and references to the original discussion thread:

https://github.com/amboar/openbmc-monorepo-discussion

Quoted 7 lines

Some of these might be closely related or considered duplicates of other list items, but based on the discussions referenced above I felt they were distinct enough to warrant separate entries.

# Desires

1. Easy sharing of a broad set of application and distro changes

+1

Quoted 1 line

2. Minimise reviews required to get a given feature or fix integrated into the distro build

+1

Quoted 1 line

3. Make fork maintenance easy

+1

Quoted 4 lines

4. Provide one place to report bugs across the entire project
5. Minimise effort collecting project statistics
6. Make it easy to add new applications
7. Make it easy to refactor across the entire project

+1

Quoted 1 line

8. Support inclusive naming

+1

Quoted 9 lines

# Problems

1. Yocto is hard
1. Managing patch stacks is hard
2. Yocto-specific tooling is hard
3. Finding the right recipe file to inspect/modify is hard
4. Yocto has too much documentation
2. OpenBMC has too much documentation

🙂 Really?

Quoted 1 line

3. Querying design/implementation/bug properties across the project is hard

+1

Quoted 1 line

4. Coordinating breaking changes is hard

+1

Quoted 1 line

5. Coordinating tree-wide changes is hard

+1

Quoted 11 lines

6. Identifying the right repo to file a bug against is hard
7. Transferring bugs between repos is hard
8. Bug reports are duplicated across repos
9. Bug reports are ignored
10. Working out where to submit a patch is hard
11. Getting patches reviewed is hard
12. New repo requests are bottle-necked

Cheers,

Andrew

#2 lore raw
Quoted 6 lines · Brad Bishop

Hi Andrew, thanks for poking at this.

Quoted 3 lines · Andrew Jeffery

# Problems

Quoted 9 lines
Quoted 6 lines

1. Yocto is hard
1. Managing patch stacks is hard
2. Yocto-specific tooling is hard
3. Finding the right recipe file to inspect/modify is hard
4. Yocto has too much documentation
2. OpenBMC has too much documentation

🙂 Really?

Hey, I'm just the messenger 😅

But of course, I could have misinterpreted the conversation. The mind map has all the quotes and reference links.

I should have mentioned that the mind map I linked to on github was generated with Freemind in case anyone wants to see something more sensible than raw XML.

Andrew

#3 lore raw
In reply to Brad Bishop
Quoted 5 lines · Brad Bishop
Quoted 2 lines

2. OpenBMC has too much documentation

🙂 Really?

I suspect this is an overgeneralization of what I intended to say. OpenBMC has enough documentation to expect that new people on the project would not be able to read all of it cover to cover before pushing a patch, and to expect that every workflow is just a matter of "go document that" or "Write a script to do that" overcomplicates things for new folks. This is not to say documentation is bad, or "too much", but the organization to be able to find relevant information quickly, and the "correctness" could certainly be improved (I fully realize it's a difficult problem). In the context of monorepo, my point ws that less day-to-day developer documentation is required, because we fall back to gits documentation, given we'd be using it in a much simpler way for the end developer.

#4 lore raw
In reply to Andrew Jeffery

Hi Andrew!

27.07.2022 04:22, Andrew Jeffery пишет:

Quoted 18 lines

# Problems

1. Yocto is hard
1. Managing patch stacks is hard
2. Yocto-specific tooling is hard
3. Finding the right recipe file to inspect/modify is hard
4. Yocto has too much documentation
2. OpenBMC has too much documentation
3. Querying design/implementation/bug properties across the project is hard
4. Coordinating breaking changes is hard
5. Coordinating tree-wide changes is hard
6. Identifying the right repo to file a bug against is hard
7. Transferring bugs between repos is hard
8. Bug reports are duplicated across repos
9. Bug reports are ignored
10. Working out where to submit a patch is hard
11. Getting patches reviewed is hard
12. New repo requests are bottle-necked

To the list of the problems I would add:

13. D-Bus is hard for newcomers not familiar with it, best practices are not described, inter-process synchronization when accessing large d-bus objects (like network interfaces) is not inherent to d-bus, and auxiliary synchronization using standard POSIX means is neither explicitly requested/advised, nor controlled/enforced. All that may lead (and have previously led) to races and various other problems. Add to that the long and inconvenient prefixing that we've discussed earlier in another thread where Brad has supported my point of those being useless to the project.

14. D-Bus may become a bottleneck or a slowing factor (due to the context switching overhead) for the situations when two processes are communicating actively. A standard POSIX IPC like pipes, mq or shm could become a solution (with d-bus or any other method used as an aid to negotiate names, keys, or whatever other credentials needed to access a common IPC).

WBR, Alexander

P.S. All in all, I think d-bus wasn't a good choice of IPC for a system running on a low-performance single-core ARM chip.

#5 lore raw
Quoted 40 lines · Alexander Amelkin

Hi Andrew!

27.07.2022 04:22, Andrew Jeffery пишет:

Quoted 18 lines

# Problems

1. Yocto is hard
1. Managing patch stacks is hard
2. Yocto-specific tooling is hard
3. Finding the right recipe file to inspect/modify is hard
4. Yocto has too much documentation
2. OpenBMC has too much documentation
3. Querying design/implementation/bug properties across the project is hard
4. Coordinating breaking changes is hard
5. Coordinating tree-wide changes is hard
6. Identifying the right repo to file a bug against is hard
7. Transferring bugs between repos is hard
8. Bug reports are duplicated across repos
9. Bug reports are ignored
10. Working out where to submit a patch is hard
11. Getting patches reviewed is hard
12. New repo requests are bottle-necked

To the list of the problems I would add:

13. D-Bus is hard for newcomers not familiar with it, best practices are not described,
inter-process synchronization when accessing large d-bus objects (like network interfaces)
is not inherent to d-bus, and auxiliary synchronization using standard POSIX means is neither
explicitly requested/advised, nor controlled/enforced.
All that may lead
(and have previously led) to
races and various other problems. Add to that the long and inconvenient prefixing that we've
discussed earlier in another thread where Brad has supported my point of those being useless
to the project.

Quoted 7 lines

14. D-Bus may become a bottleneck or a slowing factor (due to the context switching overhead) for
the situations when two processes are communicating actively. A standard POSIX IPC like pipes,
mq or shm could become a solution (with d-bus or any other method used as an aid to negotiate
names, keys, or whatever other credentials needed to access a common IPC).

FWIW, in the original context of "a single repo would help with these things" I don't think either of these would be helped with a rearrangement of code.

With that said, lots of people dislike dbus. There are performance critical things (SOL, KVM, Virtual media) that have completely avoided it. If you have a proposal for something better, I'd highly recommend writing up a design doc.

Quoted 6 lines

WBR, Alexander

P.S. All in all, I think d-bus wasn't a good choice of IPC for a system running on a low-performance single-core ARM chip.

So propose something better that solves the same problem within OpenBMC? Or, alternatively, there's u-bmc that from the sounds of reading your above is closer to your ideal in terms of trade offs (no, IPC, efficient point to point comms with grpc), it might be worth looking into for either using directly, or porting some of the ideas it encompases into OpenBMC.

#6 lore raw
Quoted 22 lines
Quoted 15 lines

13. D-Bus is hard for newcomers not familiar with it, best practices are not described,
inter-process synchronization when accessing large d-bus objects (like network interfaces)
is not inherent to d-bus, and auxiliary synchronization using standard POSIX means is neither
explicitly requested/advised, nor controlled/enforced.
All that may lead
(and have previously led) to
races and various other problems. Add to that the long and inconvenient prefixing that we've
discussed earlier in another thread where Brad has supported my point of those being useless
to the project.

Quoted 7 lines

14. D-Bus may become a bottleneck or a slowing factor (due to the context switching overhead) for
the situations when two processes are communicating actively. A standard POSIX IPC like pipes,
mq or shm could become a solution (with d-bus or any other method used as an aid to negotiate
names, keys, or whatever other credentials needed to access a common IPC).

Quoted 3 lines

FWIW, in the original context of "a single repo would help with these things" I don't think either of these would be helped with a rearrangement of code.

Well, I don't think a single repo would help actually if we want to keep the system modular and versatile as it is (and what is definitely good).

Quoted 4 lines

With that said, lots of people dislike dbus. There are performance critical things (SOL, KVM, Virtual media) that have completely avoided it. If you have a proposal for something better, I'd highly recommend writing up a design doc.

Yes, you're right that I'm not currently proposing anything specific, and that it would surely require a start with a design document.

However, I thought the idea of Andrew's message was to outline the problems, and it looks to me like you more or less agree that those are problems, don't you?

Quoted 5 lines

Or, alternatively, there's u-bmc that from the sounds of reading your above is closer to your ideal in terms of trade offs (no, IPC, efficient point to point comms with grpc), it might be worth looking into for either using directly, or porting some of the ideas it encompases into OpenBMC.

Thanks for the hint, I will look into that.

WBR, Alexander.

P.S. I feel like I must apologize for maybe sounding rude or somehow inappropriate sometimes. As you may know, English is not my native language and I, despite the vocabulary and the ability to construct more-or-less proper sentences, may often lack the ability to properly/politely express my ideas or understand the tone and/or subtle meaning of what other people say or write. Please forgive me for that, no offense or disrespect is ever implied on my side.

#7 lore raw
In reply to Ed Tanous
Quoted 10 lines · Ed Tanous
Quoted 8 lines · Alexander Amelkin

14. D-Bus may become a bottleneck or a slowing factor (due to the context switching overhead) for
the situations when two processes are communicating actively. A standard POSIX IPC like pipes,
mq or shm could become a solution (with d-bus or any other method used as an aid to negotiate
names, keys, or whatever other credentials needed to access a common IPC).

Quoted 8 lines

FWIW, in the original context of "a single repo would help with these things" I don't think either of these would be helped with a rearrangement of code.

With that said, lots of people dislike dbus. There are performance critical things (SOL, KVM, Virtual media) that have completely avoided it. If you have a proposal for something better, I'd highly recommend writing up a design doc.

Yep, fixing this requires addressing specific concerns. Patches or design docs would be helpful, or even just a list of specific things that we think are a concern.

Quoted 6 lines
Quoted 6 lines

WBR, Alexander

P.S. All in all, I think d-bus wasn't a good choice of IPC for a system running on a low-performance single-core ARM chip.

u-bmc might be something to look at here?

Andrew

#8 lore raw
In reply to Alexander Amelkin
Quoted 39 lines · Alexander Amelkin

Hi Andrew!

27.07.2022 04:22, Andrew Jeffery пишет:

Quoted 18 lines

# Problems

1. Yocto is hard
1. Managing patch stacks is hard
2. Yocto-specific tooling is hard
3. Finding the right recipe file to inspect/modify is hard
4. Yocto has too much documentation
2. OpenBMC has too much documentation
3. Querying design/implementation/bug properties across the project is hard
4. Coordinating breaking changes is hard
5. Coordinating tree-wide changes is hard
6. Identifying the right repo to file a bug against is hard
7. Transferring bugs between repos is hard
8. Bug reports are duplicated across repos
9. Bug reports are ignored
10. Working out where to submit a patch is hard
11. Getting patches reviewed is hard
12. New repo requests are bottle-necked

To the list of the problems I would add:

13. D-Bus is hard for newcomers not familiar with it, best practices are not described,
inter-process synchronization when accessing large d-bus objects (like network interfaces)
is not inherent to d-bus, and auxiliary synchronization using standard POSIX means is neither
explicitly requested/advised, nor controlled/enforced. All that may lead (and have previously led) to
races and various other problems.
Add to that the long and inconvenient
prefixing that we've
discussed earlier in another thread where Brad has supported my point of those being useless
to the project.

Quoted 7 lines

14. D-Bus may become a bottleneck or a slowing factor (due to the context switching overhead) for
the situations when two processes are communicating actively. A standard POSIX IPC like pipes,
mq or shm could become a solution (with d-bus or any other method used as an aid to negotiate
names, keys, or whatever other credentials needed to access a common IPC).

So to confirm I've understood your concerns and to distill it down to specific things people can agree or disagree with, I have:

* DBus concepts are hard
* Designing for DBus is hard
* DBus use is awkward
* DBus is slow
* Unconventional, OpenBMC-specific DBus patterns are mostly tribal knowledge

Is that fair?

Andrew

#9 lore raw
In reply to Andrew Jeffery

On 2022-07-27 09:22, Andrew Jeffery wrote> # Problems

Quoted 17 lines

1. Yocto is hard
1. Managing patch stacks is hard
2. Yocto-specific tooling is hard
3. Finding the right recipe file to inspect/modify is hard
4. Yocto has too much documentation
2. OpenBMC has too much documentation
3. Querying design/implementation/bug properties across the project is hard
4. Coordinating breaking changes is hard
5. Coordinating tree-wide changes is hard
6. Identifying the right repo to file a bug against is hard
7. Transferring bugs between repos is hard
8. Bug reports are duplicated across repos
9. Bug reports are ignored
10. Working out where to submit a patch is hard
11. Getting patches reviewed is hard
12. New repo requests are bottle-necked

I'd personally like to add one item

* Submitting patches across multiple repos is hard, especially when they depends on each other