Notes from project governance meeting at DPDK Userspace

Hi everyone,

I took some notes from a discussion we had at the end of DPDK Userspace in Dublin, concerning the growth and project structure for DPDK. If I missed anyone's name, I apologise - there were many active contributors, including most prominently Venky, Jim St Leger, Bruce Richardson, Stephen Hemminger, Chris Wright, myself, Keith Wiles, Cristian Dumitrescu, Tim O'Driscoll, Thomas Monjalon, and (until he had to leave) Vincent Jardin. There were a few others from Intel, ARM, and others, but I didn't get all the names during the discussion. If you see a comment you made and would like attribution, reply - especially if it doesn't quite match your view.

The discussion was wide ranging and we didn't quite stay on one topic until we reached a conclusion, so some of these notes are not in strict time order.

These are a mixture of notes and proposals for the project coming out of the meeting - comments are welcome, all proposals are up for discussion, and nothing has been decided on the basis of this meeting. However, all present expressed agreement that there are issues we need to address in the near future.

Apologies for the non-linear note taking, for those who were not there I hope they're useful.

Thanks,
Dave.

Topic 1: Legal entity
=====================

Do we need/want a legal entity independent of a commercial vendor who can represent the project?

Things a legal entity could do:
- Sign contracts and raise money for events
- Organise events
- Own the trademark
- Own project infrastructure like DNS, website infrastructure
- Centralised pool for marketing budget?
- Brand awareness?

There was agreement that legal governance should be lightweight, and completely independent of technical governance. Vincent insisted on the low cost structure for entities like 6WIND who would not be able to justify a 6 figure membership fee.

Topic 2: Technical governance (roadmap, project scope)
======================================================

Jim: What if soeone proposes a feature that competes with a commercial offering from an incumbent? Is it rejected, accepted, ignored?

Thomas: Issue is one of trust - is Thomas a good maintainer or not? (language moderated ;-) If we trust the maintainer, then no problem. If people disagree with Thomas as maintainer, then there are multiple ways around it - gang up on Thomas & change maintainer, or fork.

Scope is a big question - is (say) a TCP stack in scope, or not? Website says no. Thomas replies that website can be changed by a patch - patch submission would be the start of a scope discussion, and the community will decide. Venky: Scope narrowed compared to his initial goal - satisfy needs of high volume servers.

How about ODP/DPDK convergence/co-operation? (nothing major noted, beyond "the topic came up")

Do we want/need a TSC (Technical Steering Committee)? Do we need a public roadmap?

Dave: If we have a TSC, then its membership should be from the technical leadership of the project - users voices should be heard, and perhaps we should have an official user advisory group, but the technical goals and roadmap of the project should be owned by people actively developing the project.

Topic 3: Day to day technical process
=====================================

Keith raised the issue of general throughput of patches, and the number of unreviewed/wait state patches: need to actively scale the process.

Stephen: How about identifying reviewers, and Thomas designates lead reviewer for each patch? Allows scaling of review, while allowing Thomas to have last word. Alternative proposals are to have joint "maintainers" all of whom can commit patches, subject to certain constraints (don't commit your own code), or subsystem maintainers, establishing a hierarchy of trust so that Thomas can just scan and accept reviews 90% of the time.

Thomas: Need many reviewers, but there should be one committer per tree.

Stephen: Dave Miller acks minor changes, and for major changes drives discussion - no need to wait for consensus for a small, localised change. Activst maintainer means maintainer takes responsibility for converging patch discussions.
Venky: Merge windows are a bad idea - batching patch merge just make it harder to find out which patch slowed down performance - in a performance driven project you want CI & testing to track changes across individual patchsets.
Venky also pointed out the cost when you need to sync patches across multiple projects, with multi-month offsets in release dates. It can take months for a full feature to be enabled in a distro if an OpenStack patch depends on a QEMU, OVS or DPDK patch.

3 proposals from Stephen:

1. "Clone Thomas" - multiple top-level maintainers
2. Delegate - maintainer names reviewer for each patch, and trusts review
3. Subsystem maintainers - based on trust, maintainers are on the hook for their tree.

Chris: Do we have enough ppl with skills to be reviewers? If not, how do we get to that point?

Venky: Performance requires a specific skill set, not everyone has the skills, but slow patch review has an opportunity cost, and we can minimise cost of "bad" reviews with good CI performance tests that catch regressions

How do we maintain review velocity when maintainer is unavailable (eg. on vacation)? How does it happen in Linux?

- Linus delegates - names "locum" maintainer while he is away. Stephen volunteered to do it for a week or two when necessary.

Thomas: Subsystem maintainers carry review weight when maintainer is out.
Stephen: Subsystem maintainers are reviewers, need high level of trust to be a top level maintainer.

Thomas - Subtrees can live in DPDK git tree, makes it easier to manage merges afterwards. 2 volunteer subtree maintainers - Declan for crypto, Kevin for (not noted). Is it an issue if only Intel people are subtree maintainers? Venky: Yes, should have others.

Discussion summary
==================

At this point, we synthesised the conversation and tried to converge on a firm proposal for community discussion & action:

- Legal entity:
- Propose moving DPDK to Linux Foundation, with low overhead option
- Minimal governance, event, marketing budget
- Legal governance around project name, trademark

- Project leadership (roadmap, scope)
- Venky to write a proposal for broadening scope as a patch to the website
- Thomas proposes several smaller projects, rather than one umbrella
project as scope broadens
- Some discussion around whether we should limit to one project per
stack? Venky: Prefers not to
- Chris: How do decisions on new projects get made?
- Stephen: Proposed that in the event of contention, in-person debate
and consensus decides
- Jim proposed documenting current decision process, and improving on
it - documenting it will help make it better.

- Proposed criteria which should be met by any technical governance model:
1. Everyone has a voice
2. Some voices carry more weight than others, based on technical
seniority and participation in the community
3. Decisions should be time bound - after community debate, decision
should converge quickly one way or the other to give clarity

Further discussion on the composition and process for choosing a TSC:
=====================================================================

- Why not just have all the maintainers act as the TSC?
- Too many, but not open enough if community voice is not heard.
Maintainers may be tightly focused on one area, and not have overall visibility of the project.

Bruce: We should have a TSC. Elected by contributors, with some minimum bar for voting rights & being a candidate (eg. 10 patches)

Christian: Thinks it makes sense to first define scope of what a TSC would own, before deciding on a format and process for voting

- Bruce: Anything that doesn't reach community consensus gets kicked to TSC
- Cristian: Better to only have TSC decide on things which do not have
a clear decision path elsewhere (don't 2nd guess maintainer decisions)

Dave: Recommended that TSC stay small (4-5 people)

Tim will resurface his TSC proposal, revised based on this discussion, for community review and comment.

Stable/long-term releases
=========================

We finished the discussion with a brief conversation on whether we should maintain certain branches for a long time.

Bruce: Adds a lot of overhead
Venky: Can do point releases on some branches
Stephen: Prepared to "volunteer" someone from Brocade, because they need to maintain an older release anyway.

Proposal: How many parallel long-term branches?
- One per year (roughly one in 3 releases)
- 2 years maintenance after release (3 maintained branches in parallel
- 2 long-term, plus trunk)

Thanks,
Dave.

Dave Neary - NFV/SDN Community Strategy
Open Source and Standards, Red Hat - http://community.redhat.com
Ph: +1-978-399-2182 / Cell: +1-978-799-3338

Thanks Dave for the good summary.
Some comments below.

2015-10-11 01:36, Dave Neary:

Some maintainers are already identified in this file:
http://dpdk.org/browse/dpdk/tree/MAINTAINERS
It is expected that the maintainers review every patches touching their area (with exceptions for minor changes). They should also participate in discussions to get consensus.
Reviews from non-maintainers are also very welcome. Important features and fixes which are not properly reviewed should not be integrated.
Someone in the room said it is the role of the patch author to get a review. I agree.
Until now, patches from maintainers were accepted even if not reviewed. Should it be changed?

In order to scale, some maintainers could become committers of a sub-tree. The granularity of the sub-trees can evolve with growth of the project. It seems a good idea to start with trees for drivers/net, drivers/crypto and packet framework. It is not clear yet how EAL and core libraries can benefit from any sub-tree.

Contributors must be motivated enough to get some time and, in some cases, to get the green light from their management.
Some private discussions revealed that the lack of involvement is often due to some management issues.

Yes having a good distributed CI will help. But it cannot replace reviews for the quality and design considerations.

Thanks Stephen.

Having a -next tree is another way of basing its work on an up-to-date tree without waiting reviews.

About company sharing: it would be very nice to have more patches reviewed by someone working for a different company than the author's one. Considering making at least one review, when waiting for integration of its patch, would be a good start.

My understanding is that some new projects/layers can be added under the dpdk.org umbrella. So different implementations of a DPDK layer will be hosted in different dpdk.org git trees. But some implementations cannot move there, e.g. ovs-dpdk.
What about https://01.org/spdk?
http://www.seastar-project.org?
https://github.com/rumpkernel/drv-netif-dpdk?
https://01.org/intel-data-plane-performance-demonstrators?

1.8.0, 2.0.0, 2.1.0, the last digit can be used for stable branches.

You just need to send me a SSH key and the version to maintain in order to have a dedicated git tree.
Then the fixes should be preferably backported from the main tree.

Thanks everybody for the useful discussions.

Hi,

To explicitly call out the proposals and action items from the meeting:

- Legal entity proposal:
- PROPOSAL: Chris proposes moving DPDK to Linux Foundation, with low
overhead option
- Minimal governance, event, marketing budget
- Legal governance around project name, trademark

- Project leadership proposal (roadmap, scope)
- ACTION: Venky to write a proposal for broadening scope as a patch
to the website
- PROPOSAL: Thomas proposes several smaller projects, rather than one
umbrella project as scope broadens
- ACTION: Jim proposed documenting current decision process, and
improving on it - documenting it will help make it better.
- ACTION: Tim proposed to resurface his TSC proposal and drive it to
agreement and action
- Proposed criteria which should be met by any technical governance
model:
1. Everyone has a voice
2. Some voices carry more weight than others, based on technical
seniority and participation in the community
3. Decisions should be time bound - after community debate, decision
should converge quickly one way or the other to give clarity

- Day to day patch review:
- PROPOSAL: Thomas: Create hierarchical review process with
maintainers responsible for sub-trees (to be housed in DPDK Git)
- ACTION: (without owner?) Subtrees and maintainers to be identified,
-next, crypto and (drivers, IIRC?) to be first trees
- PROPOSAL: Thomas to identify replacement maintainers short-term
when he is unable to do it (vacation, sickness, etc)

- Stable patch maintenance
- PROPOSAL: Maintain one release per year as a long term release,
with point releases being made regularly (based on patch volume), with branches maintained for 2 years (2 stable branches + 1 devel branch active at all times).

In addition to Thomas's notes, does this cover all of the conclusions that came out of the meeting?

Thanks,
Dave.

Dave Neary - NFV/SDN Community Strategy
Open Source and Standards, Red Hat - http://community.redhat.com
Ph: +1-978-399-2182 / Cell: +1-978-799-3338