Skip to contentSkip to conversation

KVM updates for the 3.4 merge window

GIT PULL Avi Kivity 6 participants 27 messages Mar 20, 2012 – Apr 17, 2012 KVM

lore
Share & tools
lore

Avi Kivity

author
#0 lore raw

Linus, please pull from

ra.kernel.org:/pub/scm/virt/kvm/kvm.git kvm-updates/3.4

(ssh URL as git.kernel.org is down at the moment) to receive the KVM updates for the 3.4 merge window. Changes include timekeeping improvements, support for assigning host PCI devices that share interrupt lines, s390 user-controlled guests, a large ppc update, and random fixes.

----------------------------------------------------------------
Alex Shi (1):
KVM: use correct tlbs dirty type in cmpxchg

Alexander Graf (13):
KVM: PPC: E500: Support hugetlbfs
KVM: PPC: Book3s: PR: Disable preemption in vcpu_run
KVM: PPC: Book3s: PR: No irq_disable in vcpu_run
KVM: PPC: Use get/set for to_svcpu to help preemption
KVM: PPC: align vcpu_kick with x86
KVM: PPC: Book3S: PR: Fix signal check race
KVM: PPC: Add generic single register ioctls
KVM: PPC: Add support for explicit HIOR setting
KVM: PPC: Rename MMIO register identifiers
KVM: PPC: E500: Fail init when not on e500v2
KVM: PPC: Convert RMA allocation into generic code
KVM: PPC: Initialize linears with zeros
KVM: PPC: Add HPT preallocator

Avi Kivity (5):
KVM: x86 emulator: add 8-bit memory operands
KVM: x86 emulator: Remove byte-sized MOVSX/MOVZX hack
KVM: x86 emulator: reject SYSENTER in compatibility mode on AMD guests
KVM: Ensure all vcpus are consistent with in-kernel irqchip settings
KVM: VMX: Fix delayed load of shared MSRs

Bharat Bhushan (3):
PPC: Fix race in mtmsr paravirt implementation
KVM: PPC: Fix DEC truncation for greater than 0xffff_ffff/1000
KVM: PPC: booke: Do Not start decrementer when SPRN_DEC set 0

Boris Ostrovsky (1):
KVM: SVM: Add support for AMD's OSVW feature in guests

Carsten Otte (11):
KVM: s390: add parameter for KVM_CREATE_VM
KVM: s390: ucontrol: per vcpu address spaces
KVM: s390: ucontrol: export page faults to user
KVM: s390: ucontrol: export SIE control block to user
KVM: s390: ucontrol: disable in-kernel handling of SIE intercepts
KVM: s390: ucontrol: disable in-kernel irq stack
KVM: s390: ucontrol: interface to inject faults on a vcpu page table
KVM: s390: ucontrol: disable sca
KVM: s390: fix assumption for KVM_MAX_VCPUS
KVM: s390: ucontrol: announce capability for user controlled vms
KVM: s390: Fix return code for unknown ioctl numbers

Christian Borntraeger (7):
KVM: s390: rework code that sets the prefix
KVM: provide synchronous registers in kvm_run
KVM: s390: provide the prefix register via kvm_run
KVM: s390: provide general purpose guest registers via kvm_run
KVM: s390: provide access guest registers via kvm_run
KVM: s390: Sanitize fpc registers for KVM_SET_FPU
KVM: s390: provide control registers via kvm_run

Danny Kukawka (1):
arch/powerpc/kvm/book3s_hv.c: included linux/sched.h twice

Davidlohr Bueso (3):
KVM: MMU: unnecessary NX state assignment
KVM: SVM: comment nested paging and virtualization module parameters
KVM: MMU: make use of ->root_level in reset_rsvds_bits_mask

Gleb Natapov (5):
KVM: x86: reset edge sense circuit of i8259 on init
KVM: x86 emulator: correctly mask pmc index bits in RDPMC
instruction emulation
KVM: PMU: warn when pin control is set in eventsel msr
KVM: PMU: Fix raw event check
KVM: PMU: add proper support for fixed counter 2

Igor Mammedov (1):
x86: Introduce x86_cpuinit.early_percpu_clock_init hook

Jan Kiszka (2):
KVM: Allow host IRQ sharing for assigned PCI 2.3 devices
KVM: Convert intx_mask_lock to spin lock

Jens Freimann (4):
KVM: s390: do store status after handling STOP_ON_STOP bit
KVM: s390: make sigp restart return busy when stop pending
KVM: s390: ignore sigp stop overinitiative
KVM: s390: add stop_on_stop flag when doing stop and store

Julian Stecklina (1):
KVM: Don't mistreat edge-triggered INIT IPI as INIT de-assert. (LAPIC)

Kevin Wolf (4):
KVM: x86 emulator: Fix task switch privilege checks
KVM: x86 emulator: VM86 segments must have DPL 3
KVM: SVM: Fix CPL updates
KVM: x86 emulator: Allow PM/VM86 switch during task switch

Liu Yu (1):
KVM: PPC: booke: Add booke206 TLB trace

Liu Yu-B13201 (1):
KVM: PPC: Avoid patching paravirt template code

Marcelo Tosatti (4):
KVM: x86: increase recommended max vcpus to 160
KVM: Allow adjust_tsc_offset to be in host or guest cycles
x86: kvmclock: abstract save/restore sched_clock_state
KVM: x86: fix kvm_write_tsc() TSC matching thinko

Matt Evans (2):
KVM: PPC: Fix vcpu_create dereference before validity check.
KVM: PPC: Add KVM_CAP_NR_VCPUS and KVM_CAP_MAX_VCPUS

Michael S. Tsirkin (1):
KVM: fix error handling for out of range irq

Nadav Har'El (1):
KVM: nVMX: Fix erroneous exception bitmap check

Nicolae Mogoreanu (1):
KVM: Ignore the writes to MSR_K7_HWCR(3)

Paul Mackerras (19):
KVM: PPC: Make wakeups work again for Book3S HV guests
KVM: PPC: Keep a record of HV guest view of hashed page table entries
KVM: PPC: Keep page physical addresses in per-slot arrays
KVM: PPC: Add an interface for pinning guest pages in Book3s HV guests
KVM: PPC: Make the H_ENTER hcall more reliable
KVM: PPC: Only get pages when actually needed, not in
prepare_memory_region()
KVM: PPC: Allow use of small pages to back Book3S HV guests
KVM: PPC: Allow I/O mappings in memory slots
KVM: PPC: Maintain a doubly-linked list of guest HPTEs for each gfn
KVM: PPC: Implement MMIO emulation support for Book3S HV guests
KVM: PPC: Implement MMU notifiers for Book3S HV guests
KVM: Add barriers to allow mmu_notifier_retry to be used locklessly
KVM: PPC: Allow for read-only pages backing a Book3S HV guest
KVM: PPC: Book3S HV: Keep HPTE locked when invalidating
KVM: PPC: Book3s HV: Maintain separate guest and host views of R
and C bits
KVM: PPC: Book3S HV: Use the hardware referenced bit for kvm_age_hva
KVM: PPC: Book3s HV: Implement get_dirty_log using hardware
changed bit
KVM: PPC: Move kvm_vcpu_ioctl_[gs]et_one_reg down to
platform-specific code
KVM: Move gfn_to_memslot() to kvm_host.h

Raghavendra K T (1):
KVM: VMX: remove yield_on_hlt

Sasha Levin (1):
KVM: PPC: Use the vcpu kmem_cache when allocating new VCPUs

Scott Wood (17):
KVM: PPC: e500: don't translate gfn to pfn with preemption disabled
KVM: PPC: e500: Eliminate preempt_disable in local_sid_destroy_all
KVM: PPC: e500: clear up confusion between host and guest entries
KVM: PPC: e500: MMU API
KVM: PPC: e500: tlbsx: fix tlb0 esel
KVM: PPC: e500: Don't hardcode PIR=0
KVM: PPC: booke: check for signals in kvmppc_vcpu_run
KVM: PPC: Rename deliver_interrupts to prepare_to_enter
KVM: PPC: Move prepare_to_enter call site into subarch code
KVM: PPC: booke: Check for MSR[WE] in prepare_to_enter
KVM: PPC: booke: Fix int_pending calculation for MSR[EE] paravirt
KVM: PPC: booke: Paravirtualize wrtee
KVM: PPC: Paravirtualize SPRG4-7, ESR, PIR, MASn
KVM: PPC: booke: Improve timer register emulation
KVM: PPC: e500: Fix TLBnCFG in KVM_CONFIG_TLB
KVM: PPC: e500: use hardware hint when loading TLB0 entries
KVM: PPC: refer to paravirt docs in header file

Takuya Yoshikawa (11):
KVM: MMU: Remove for_each_unsync_children() macro
KVM: MMU: Add missing large page accounting to drop_large_spte()
KVM: MMU: Remove unused kvm_pte_chain
KVM: MMU: Remove unused kvm parameter from __gfn_to_rmap()
KVM: MMU: Remove unused kvm parameter from rmap_next()
KVM: Fix write protection race during dirty logging
KVM: Introduce gfn_to_index() which returns the index for a given
level
KVM: Split lpage_info creation out from __kvm_set_memory_region()
KVM: Simplify ifndef conditional usage in __kvm_set_memory_region()
KVM: Introduce kvm_memory_slot::arch and move lpage_info into it
KVM: mmu_notifier: Flush TLBs before releasing mmu_lock

Xiao Guangrong (1):
KVM: MMU: remove the redundant get_written_sptes

Zachary Amsden (7):
KVM: Infrastructure for software and hardware based TSC rate scaling
KVM: Improve TSC offset matching
KVM: Leave TSC synchronization window open with each new sync
KVM: Fix last_guest_tsc / tsc_offset semantics
KVM: Add last_host_tsc tracking back to KVM
KVM: Dont mark TSC unstable due to S4 suspend
KVM: Track TSC synchronization in generations

 Documentation/virtual/kvm/api.txt        |  259 +++++++++-
 Documentation/virtual/kvm/ppc-pv.txt     |   24 +-
 arch/ia64/include/asm/kvm.h              |    4 +
 arch/ia64/include/asm/kvm_host.h         |    3 +
 arch/ia64/kvm/kvm-ia64.c                 |   25 +-
 arch/powerpc/include/asm/kvm.h           |   46 ++-
 arch/powerpc/include/asm/kvm_book3s.h    |   98 +++-
 arch/powerpc/include/asm/kvm_book3s_32.h |    6 +-
 arch/powerpc/include/asm/kvm_book3s_64.h |  180 ++++++-
 arch/powerpc/include/asm/kvm_e500.h      |   52 +-
 arch/powerpc/include/asm/kvm_host.h      |   90 +++-
 arch/powerpc/include/asm/kvm_para.h      |   41 ++-
 arch/powerpc/include/asm/kvm_ppc.h       |   25 +-
 arch/powerpc/include/asm/mmu-book3e.h    |    6 +-
 arch/powerpc/include/asm/mmu-hash64.h    |    2 +-
 arch/powerpc/include/asm/ppc-opcode.h    |    4 +-
 arch/powerpc/include/asm/reg.h           |    5 +
 arch/powerpc/kernel/asm-offsets.c        |   16 +-
 arch/powerpc/kernel/exceptions-64s.S     |    8 +-
 arch/powerpc/kernel/kvm.c                |  307 +++++++++--
 arch/powerpc/kernel/kvm_emul.S           |  112 +++-
 arch/powerpc/kernel/setup_64.c           |    2 +-
 arch/powerpc/kvm/Kconfig                 |    1 +
 arch/powerpc/kvm/book3s.c                |   57 +--
 arch/powerpc/kvm/book3s_32_mmu_host.c    |   21 +-
 arch/powerpc/kvm/book3s_64_mmu_host.c    |   66 ++-
 arch/powerpc/kvm/book3s_64_mmu_hv.c      |  919
++++++++++++++++++++++++++++--
 arch/powerpc/kvm/book3s_emulate.c        |    8 +-
 arch/powerpc/kvm/book3s_hv.c             |  466 ++++++++++------
 arch/powerpc/kvm/book3s_hv_builtin.c     |  209 +++++---
 arch/powerpc/kvm/book3s_hv_rm_mmu.c      |  835 +++++++++++++++++++++------
 arch/powerpc/kvm/book3s_hv_rmhandlers.S  |  176 +++++-
 arch/powerpc/kvm/book3s_paired_singles.c |    9 +-
 arch/powerpc/kvm/book3s_pr.c             |  178 +++++-
 arch/powerpc/kvm/booke.c                 |  150 ++++--
 arch/powerpc/kvm/booke.h                 |    4 +
 arch/powerpc/kvm/booke_emulate.c         |   23 +-
 arch/powerpc/kvm/booke_interrupts.S      |   18 +-
 arch/powerpc/kvm/e500.c                  |   32 +-
 arch/powerpc/kvm/e500_emulate.c          |   38 +-
 arch/powerpc/kvm/e500_tlb.c              |  775 ++++++++++++++++++--------
 arch/powerpc/kvm/e500_tlb.h              |   80 +--
 arch/powerpc/kvm/emulate.c               |   61 +-
 arch/powerpc/kvm/powerpc.c               |  148 ++++--
 arch/powerpc/kvm/trace.h                 |   62 ++-
 arch/powerpc/mm/hugetlbpage.c            |    2 +
 arch/s390/include/asm/kvm.h              |   11 +
 arch/s390/include/asm/kvm_host.h         |   12 +-
 arch/s390/kvm/Kconfig                    |    9 +
 arch/s390/kvm/diag.c                     |    6 +-
 arch/s390/kvm/intercept.c                |   24 +-
 arch/s390/kvm/interrupt.c                |    3 +-
 arch/s390/kvm/kvm-s390.c                 |  221 ++++++--
 arch/s390/kvm/kvm-s390.h                 |   18 +
 arch/s390/kvm/priv.c                     |   27 +-
 arch/s390/kvm/sigp.c                     |   57 ++-
 arch/x86/include/asm/kvm.h               |    4 +
 arch/x86/include/asm/kvm_emulate.h       |    3 +-
 arch/x86/include/asm/kvm_host.h          |   63 ++-
 arch/x86/include/asm/perf_event.h        |    1 +
 arch/x86/include/asm/tsc.h               |    4 +-
 arch/x86/include/asm/x86_init.h          |    6 +
 arch/x86/kernel/kvmclock.c               |   15 +-
 arch/x86/kernel/smpboot.c                |    1 +
 arch/x86/kernel/tsc.c                    |    4 +-
 arch/x86/kernel/x86_init.c               |    5 +-
 arch/x86/kvm/cpuid.c                     |    2 +-
 arch/x86/kvm/cpuid.h                     |    8 +
 arch/x86/kvm/emulate.c                   |  112 ++++-
 arch/x86/kvm/i8259.c                     |    1 +
 arch/x86/kvm/lapic.c                     |    4 +-
 arch/x86/kvm/mmu.c                       |   85 ++--
 arch/x86/kvm/mmu_audit.c                 |    4 +-
 arch/x86/kvm/pmu.c                       |   10 +-
 arch/x86/kvm/svm.c                       |  119 ++++-
 arch/x86/kvm/vmx.c                       |   53 +-
 arch/x86/kvm/x86.c                       |  403 ++++++++++---
 arch/x86/power/cpu.c                     |    4 +-
 include/linux/kvm.h                      |   98 ++++
 include/linux/kvm_host.h                 |   69 ++-
 virt/kvm/assigned-dev.c                  |  213 ++++++-
 virt/kvm/kvm_main.c                      |  144 ++----
 82 files changed, 5808 insertions(+), 1668 deletions(-)
Signature · 1 line
error compiling committee.c: too many arguments to function
#1 lore raw
Quoted 9 lines · Avi Kivity

Linus, please pull from

ra.kernel.org:/pub/scm/virt/kvm/kvm.git kvm-updates/3.4

(ssh URL as git.kernel.org is down at the moment) to receive the KVM updates for the 3.4 merge window. Changes include timekeeping improvements, support for assigning host PCI devices that share interrupt lines, s390 user-controlled guests, a large ppc update, and random fixes.

I wonder if Linus missed this one ... :-)

But that's not the point of the email ... Avi, I have a bit of a problem with the way you manage your tree(s).

It appears to me that you have on one side, a tree that has all of KVM dev. history for years, which is used as a parent by your sub-maintainers such as Alex, and as your main dev. tree. However, that -tree- is not related to Linus, and whenever you do a pull request, you basically rebase patches on top of a different tree which is derived from Linus.

That means that everything gets constantly rebased, and it makes life very much harder for us working with this.

For our own dev, we have to essentially work on trees that derive from Linus, my own -next branch, Alex's -next branch, whatever you have going on etc... and updating/rebasing our own local WIP patches is extremely painful because there is never a recent common ancestor between your trees (and other deriving from it such as Alex) and anything else.

Is there any reason why you keep working that way other than a bad habit ? :-)

It would be much easier for everybody involved if your WIP development was always based on a recent ancestor and that you did not rebase it, meaning that Linus would just "pull" it and everybody would resync cleanly after every merge window.

Cheers,
Ben.

#2 lore raw
Quoted 3 lines · Benjamin Herrenschmidt

That means that everything gets constantly rebased, and it makes life very much harder for us working with this.

Ben, thanks for pointing this out.

I will not be pulling this tree at all. It's pure and utter shit, and I wonder how long (forever?) this has been going on.

The particular issue that upsets *me* is only indirectly the problem Ben is mentioning. No, the thing that makes me go "uhhuh, no way in *hell* should I pull this" is that you have apparently totally broken all sign-offs.

Avi, you ABSOLUTELY MUST NOT rebase other peoples commits. That's a total no-no. And one thing I notice when I look through the commits is that you have totally broken the Signed-off-by: series in the process, exactly because what you do is crap, crap, CRAP.

The sign-off chain should be very simple: the first person to sign off should be the author, and the last person to sign off should be the committer.

That's simply not true in your tree. Maybe because you have rebased other peoples (Alexander's) commits? I see commits where the sign-off ends with Alexander, but then the committer is you. WTF?

Fix your f*cking broken shit *now*.

I'm not pulling crap like this. And it makes me unhappy to realize that this has probably happened a long time and I haven't even noticed.

The whole "you MUST NOT rebase other peoples commits" is the thing I've been telling people for *years* now. Why the hell is it still going on?

Linus

author
#3 lore raw
Quoted 31 lines · Linus Torvalds
Quoted 3 lines · Benjamin Herrenschmidt

That means that everything gets constantly rebased, and it makes life very much harder for us working with this.

Ben, thanks for pointing this out.

I will not be pulling this tree at all. It's pure and utter shit, and I wonder how long (forever?) this has been going on.

The particular issue that upsets *me* is only indirectly the problem Ben is mentioning. No, the thing that makes me go "uhhuh, no way in *hell* should I pull this" is that you have apparently totally broken all sign-offs.

Avi, you ABSOLUTELY MUST NOT rebase other peoples commits. That's a total no-no. And one thing I notice when I look through the commits is that you have totally broken the Signed-off-by: series in the process, exactly because what you do is crap, crap, CRAP.

The sign-off chain should be very simple: the first person to sign off should be the author, and the last person to sign off should be the committer.

That's simply not true in your tree. Maybe because you have rebased other peoples (Alexander's) commits? I see commits where the sign-off ends with Alexander, but then the committer is you. WTF?

Fix your f*cking broken shit *now*.

Quoted 7 lines

I'm not pulling crap like this. And it makes me unhappy to realize that this has probably happened a long time and I haven't even noticed.

The whole "you MUST NOT rebase other peoples commits" is the thing I've been telling people for *years* now. Why the hell is it still going on?

Well I've been doing this ever since I moved to git. The motivation was actually to make things easier for patch authors by allowing them to work against a tree of all applied patches, while the update for the next merge window is just a subset, with more fixes going into the merge window even late in the cycle, and features being deferred to the next one. I also fold fixes or reverts into their parent patches to improve bisectability.

I can switch to fast-forward-only in the future, but I'm afraid that this particular tree is broken for good. The un-rebased fast-forward-only source for this is kvm.git master, which I don't think you want to pull. It will cause every kvm commit to appear twice and confuse everyone.

Signature · 1 line
error compiling committee.c: too many arguments to function
#4 lore raw
Quoted 8 lines · Avi Kivity

Well I've been doing this ever since I moved to git. The motivation was actually to make things easier for patch authors by allowing them to work against a tree of all applied patches, while the update for the next merge window is just a subset, with more fixes going into the merge window even late in the cycle, and features being deferred to the next one. I also fold fixes or reverts into their parent patches to improve bisectability.

Quoted 5 lines

I can switch to fast-forward-only in the future, but I'm afraid that this particular tree is broken for good. The un-rebased fast-forward-only source for this is kvm.git master, which I don't think you want to pull. It will cause every kvm commit to appear twice and confuse everyone.

The problem is that it makes it very hard if not impossible to work with a combination of your tree & other trees (for example at some point I had to work on a merge of alex'tree, powerpc-next and pci-next).

I don't see the problem with using the standard way and having sub-maintainers/developers.... Most of my sub-maintainers work on top of some upstream or they branch off my -next branch (which is known to never be rebased, so it's resync'ed as soon as Linux pulls it). Dealing with branches & merges in git is trivial and easier than dealing with the clashes caused by the rebases :-)

One thing I do, is to also sometimes put out a powerpc-test branch that people know can and will be rebased, it's purely there if I want some folks to test a bunch of stuff but without basing their own work on top of it.

And yes, there's a drawback vs. bisectability. You can still fold-in if you pickup patches from the list (vs pulling from sub-maintainers) as long as you haven't committed them to a "non-rebase" branch (ie, you can let things stage in a test branch for example for a couple of weeks to flush out those issues).

Cheers,
Ben.

author
#5 lore raw
Quoted 16 lines · Benjamin Herrenschmidt
Quoted 8 lines · Avi Kivity

Well I've been doing this ever since I moved to git. The motivation was actually to make things easier for patch authors by allowing them to work against a tree of all applied patches, while the update for the next merge window is just a subset, with more fixes going into the merge window even late in the cycle, and features being deferred to the next one. I also fold fixes or reverts into their parent patches to improve bisectability.

Quoted 5 lines

I can switch to fast-forward-only in the future, but I'm afraid that this particular tree is broken for good. The un-rebased fast-forward-only source for this is kvm.git master, which I don't think you want to pull. It will cause every kvm commit to appear twice and confuse everyone.

Quoted 8 lines

The problem is that it makes it very hard if not impossible to work with a combination of your tree & other trees (for example at some point I had to work on a merge of alex'tree, powerpc-next and pci-next).

I don't see the problem with using the standard way and having sub-maintainers/developers.... Most of my sub-maintainers work on top of some upstream or they branch off my -next branch (which is known to never be rebased, so it's resync'ed as soon as Linux pulls it)

Say a fix comes in which needs to be mainlined during -rc. So I put it in some other branch, to be sent off to Linus in a few days after maturing a little. Meanwhile developers see an incomplete tree, since that patch is not in it.

Once Linus pulls, I can merge it back (or even before, if I'm reasonably certain it's not going to change), but it leaves a history of unneeded merges. Or we can do throwaway merges like tip.git.

Quoted 9 lines

. Dealing
with branches & merges in git is trivial and easier than dealing with the clashes caused by the rebases :-)

One thing I do, is to also sometimes put out a powerpc-test branch that people know can and will be rebased, it's purely there if I want some folks to test a bunch of stuff but without basing their own work on top of it.

Quoted 5 lines

And yes, there's a drawback vs. bisectability. You can still fold-in if you pickup patches from the list (vs pulling from sub-maintainers) as long as you haven't committed them to a "non-rebase" branch (ie, you can let things stage in a test branch for example for a couple of weeks to flush out those issues).

Right, we'll probably do something along these lines.

Signature · 1 line
error compiling committee.c: too many arguments to function
#6 lore raw
Quoted 9 lines · Avi Kivity

Say a fix comes in which needs to be mainlined during -rc. So I put it in some other branch, to be sent off to Linus in a few days after maturing a little. Meanwhile developers see an incomplete tree, since that patch is not in it.

Once Linus pulls, I can merge it back (or even before, if I'm reasonably certain it's not going to change), but it leaves a history of unneeded merges. Or we can do throwaway merges like tip.git.

We don't do throwaway merges in the -tip development branches themselves, i.e. in tip:sched/core, tip:perf/core, tip:timers/core, etc.

When a fix goes into tip:sched/urgent then until Linus merges it it's not in tip:sched/core. 99% of the fixes don't *have to* go into sched/core straight away.

In the odd case where there's some dependency, we can manually merge it into tip:sched/core ahead of Linus pulling into an -rc. Those rare merges are not a problem, and I explain the reason in the merge commit itself.

If you look at:

gll v3.2..v3.3 | grep -E '/urgent.*/core'

you'll see that I only had to do it once in the previous cycle:

d6c1c49de577 Merge branch 'perf/urgent' into perf/core

and the changelog explains the background:

Merge reason: Add these cherry-picked commits so that future changes
on perf/core don't conflict.

it was a rare, oddball situation where we cherry-picked perf/core changes into perf/urgent. Extra merges are perfectly fine in that case.

The 'throwaway' tip:master branch you are probably referring to is basically just a testing branch, a convenient merged tree of the one dozen maintainer trees that are hosted in -tip. Since we don't want to force Linus's hand of him being able to reject individual trees we don't merge them properly - hence the integrated tree is a throwaway tree in theory.

In practice I tend to throw it away only once per cycle, around -rc1, once all pending trees went to Linus. tip:master is not used for any Git based contribution work - it's for testing, it's for people who want to work with patches - the commits themselves always go into persistent non-rebasing, append-only Git trees.

If we mess up bisectability we do a delta fix. When choosing between somewhat better bisectability and a proper history that others can rely on then proper history wins hands down.

Thanks,

Ingo

author
#7 lore raw
Quoted 56 lines · Ingo Molnar
Quoted 9 lines · Avi Kivity

Say a fix comes in which needs to be mainlined during -rc. So I put it in some other branch, to be sent off to Linus in a few days after maturing a little. Meanwhile developers see an incomplete tree, since that patch is not in it.

Once Linus pulls, I can merge it back (or even before, if I'm reasonably certain it's not going to change), but it leaves a history of unneeded merges. Or we can do throwaway merges like tip.git.

We don't do throwaway merges in the -tip development branches themselves, i.e. in tip:sched/core, tip:perf/core, tip:timers/core, etc.

When a fix goes into tip:sched/urgent then until Linus merges it it's not in tip:sched/core. 99% of the fixes don't *have to* go into sched/core straight away.

In the odd case where there's some dependency, we can manually merge it into tip:sched/core ahead of Linus pulling into an -rc. Those rare merges are not a problem, and I explain the reason in the merge commit itself.

If you look at:

gll v3.2..v3.3 | grep -E '/urgent.*/core'

you'll see that I only had to do it once in the previous cycle:

d6c1c49de577 Merge branch 'perf/urgent' into perf/core

and the changelog explains the background:

Merge reason: Add these cherry-picked commits so that future changes
on perf/core don't conflict.

it was a rare, oddball situation where we cherry-picked perf/core changes into perf/urgent. Extra merges are perfectly fine in that case.

The 'throwaway' tip:master branch you are probably referring to is basically just a testing branch, a convenient merged tree of the one dozen maintainer trees that are hosted in -tip. Since we don't want to force Linus's hand of him being able to reject individual trees we don't merge them properly - hence the integrated tree is a throwaway tree in theory.

In practice I tend to throw it away only once per cycle, around -rc1, once all pending trees went to Linus. tip:master is not used for any Git based contribution work - it's for testing, it's for people who want to work with patches - the commits themselves always go into persistent non-rebasing, append-only Git trees.

Quoted 4 lines

If we mess up bisectability we do a delta fix. When choosing between somewhat better bisectability and a proper history that others can rely on then proper history wins hands down.

Okay, we'll adopt a similar workflow for the future.

Signature · 1 line
error compiling committee.c: too many arguments to function
#8 lore raw
In reply to Avi Kivity
Quoted 8 lines · Avi Kivity

Say a fix comes in which needs to be mainlined during -rc. So I put it in some other branch, to be sent off to Linus in a few days after maturing a little. Meanwhile developers see an incomplete tree, since that patch is not in it.

Once Linus pulls, I can merge it back (or even before, if I'm reasonably certain it's not going to change), but it leaves a history of unneeded merges. Or we can do throwaway merges like tip.git.

So to give an example, take powerpc. There's two branches, next and merge. After the merge window, at rc1, they are basically empty.

Developers are encouraged to work on top of Linus upstream. If they depend on each other they are free to pull each other stuff in, as long as they avoid rebasing or deal with each other when that is done.

At later rc's (it should be as soon as rc1/rc2 but I trend to be a bit late there) I open powerpc-next where I start putting stuff that's intended for the next merge window. This is virtually upstream, this branch doesn't get rebased.

So developers can fork of it if they wish to, or merge it into their branch if they need some of the new work, there's really no big deal with a few spurrious merges if there's a good reason for that, git is good at it and it's pretty harmless.

Every now and then, I have fixes that I want to send Linus during -rc, in which case I put them in a "powerpc-merge" branch. This is very short lived, the fixes have generally been on the list/patchwork already, and except maybe around rc1, have to be simple enough to be fairly low risk, so they don't need that much "maturation" (besides it's not like anybody tests "merge" anyways). So I put things in there, run it through my automated build tests, do a few runtime tests and send them to Linus, sometimes even the same day, though the patches themselves will usually have been on the list & patchwork for a few days.

Now those patches don't absolutely need to be in "next". If they fix something nasty enough or potentially conflicting with work in progress, then I can just pull Linus back into next after he merges or even pull "merge" into "next" if I don't want to wait for Linus.

Another option, is to actually cherry pick some of those patches and apply them to both next and merge. This is perfectly fine, git will resolve things 99% of the time and at worst it's going to be a bit of context fixup in the final merge which is easy to deal with.

Basically that means that I have -0- history fight after a merge window. My trees essentially all fast-forward to Linus as soon as he has pulled.

My throw-away test branch is seldomly used, mostly if there's stuff that I want beaten up a bit more than usual, in which case I ask folks around to give it a go and put it up there.

Cheers,
Ben.

#9 lore raw
In reply to Avi Kivity
Quoted 5 lines · Avi Kivity

I can switch to fast-forward-only in the future, but I'm afraid that this particular tree is broken for good. The un-rebased fast-forward-only source for this is kvm.git master, which I don't think you want to pull. It will cause every kvm commit to appear twice and confuse everyone.

There are patches from me in there that have been pending since December last year, and now look like they won't be going upstream until June. So, under the circumstances, how would you (Avi) feel about Ben and I committing the KVM patches that only affect powerpc to Ben's tree and sending them to Linus that way before the merge window closes?

Paul.

author
#10 lore raw
Quoted 8 lines · Paul Mackerras
Quoted 5 lines · Avi Kivity

I can switch to fast-forward-only in the future, but I'm afraid that this particular tree is broken for good. The un-rebased fast-forward-only source for this is kvm.git master, which I don't think you want to pull. It will cause every kvm commit to appear twice and confuse everyone.

Quoted 6 lines

There are patches from me in there that have been pending since December last year, and now look like they won't be going upstream until June. So, under the circumstances, how would you (Avi) feel about Ben and I committing the KVM patches that only affect powerpc to Ben's tree and sending them to Linus that way before the merge window closes?

That's fine if there are no interdependencies. It looks like 74df956 will be a problem though.

The other two options are:

- I'll add my signoff to the commits that lack it. This unbreaks the committer lacks signoff.
- I remove Alex's commits, Alex sends me a pull request based on that tree, I pull it in. Alex being away for three weeks is a minor flaw in this plan.
- As above, but with you sending me that tree as temporary kvm-ppc maintainer.

Options 2/3 are harder as there are a few interdependencies.

Linus, if the first option is acceptable, a tree is available at the same place:

git://git.kernel.org/pub/scm/virt/kvm/kvm.git kvm-updates/3.4

otherwise, please provide guidance.

Signature · 1 line
error compiling committee.c: too many arguments to function
#11 lore raw
Quoted 13 lines · Avi Kivity

That's fine if there are no interdependencies. It looks like 74df956 will be a problem though.

The other two options are:

- I'll add my signoff to the commits that lack it. This unbreaks the committer lacks signoff.
- I remove Alex's commits, Alex sends me a pull request based on that tree, I pull it in. Alex being away for three weeks is a minor flaw in this plan.
- As above, but with you sending me that tree as temporary kvm-ppc maintainer.

Quoted 8 lines

Options 2/3 are harder as there are a few interdependencies.

Linus, if the first option is acceptable, a tree is available at the same place:

git://git.kernel.org/pub/scm/virt/kvm/kvm.git kvm-updates/3.4

otherwise, please provide guidance.

Linus, what's your feeling here ? I'd really like to have the bulk of the powerpc updates this time around, they already missed the previous merge window due to maintainers taking too long to deal with their inbox :-)

KVM ppc is a big WIP with little if any users at this point so the risk of breaking things is near to non-existent, it all boils down to making our own life easier by having stuff not drifting out of tree (and making it possible for distros to pick it up when they decide to do so).

Cheers,
Ben.

#12 lore raw
Quoted 5 lines · Benjamin Herrenschmidt

Linus, what's your feeling here ? I'd really like to have the bulk of the powerpc updates this time around, they already missed the previous merge window due to maintainers taking too long to deal with their inbox :-)

If the proper sign-offs got added, and we think that future development will be sane and orderly, I guess I'll pull.

Linus

#13 lore raw
In reply to Avi Kivity
Quoted 7 lines · Avi Kivity
Quoted 6 lines

There are patches from me in there that have been pending since December last year, and now look like they won't be going upstream until June. So, under the circumstances, how would you (Avi) feel about Ben and I committing the KVM patches that only affect powerpc to Ben's tree and sending them to Linus that way before the merge window closes?

Quoted 7 lines

That's fine if there are no interdependencies. It looks like 74df956 will be a problem though.

The other two options are:

- I'll add my signoff to the commits that lack it. This unbreaks the committer lacks signoff.

I just noticed that the branch you asked Linus to pull includes none of the patches that Alex sent you in the last batch, in the email with subject "[PULL 00/56] ppc patch queue 2012-03-15" sent on March 15, where he asked you to pull git://github.com/agraf/linux-2.6.git
for-upstream.

What happened? Did they get lost in the re-signing, or is there some reason you thought they shouldn't go in?

Thanks,
Paul.

author
#14 lore raw
Quoted 8 lines · Paul Mackerras

I just noticed that the branch you asked Linus to pull includes none of the patches that Alex sent you in the last batch, in the email with subject "[PULL 00/56] ppc patch queue 2012-03-15" sent on March 15, where he asked you to pull git://github.com/agraf/linux-2.6.git
for-upstream.

What happened? Did they get lost in the re-signing, or is there some reason you thought they shouldn't go in?

That pull request was send three days before the merge window opened; patches are supposed to cook for a while in -next before being merged, especially large trees like that one.

Signature · 1 line
error compiling committee.c: too many arguments to function
#15 lore raw
Quoted 7 lines · Avi Kivity
Quoted 6 lines · Paul Mackerras

I just noticed that the branch you asked Linus to pull includes none of the patches that Alex sent you in the last batch, in the email with subject "[PULL 00/56] ppc patch queue 2012-03-15" sent on March 15, where he asked you to pull git://github.com/agraf/linux-2.6.git
for-upstream.

Quoted 6 lines
Quoted 2 lines

What happened? Did they get lost in the re-signing, or is there some reason you thought they shouldn't go in?

That pull request was send three days before the merge window opened; patches are supposed to cook for a while in -next before being merged, especially large trees like that one.

These are all powerpc specific patches that have been cooking in Alex tree for a while and elsewhere before that. They almost only affect arch/powerpc/kvm, and as such don't really need a lot of integration testing in -next. A bit for sure but not necessarily monthes.

The current process is such that it takes absolutely forever for our patches to get in, which is a major PITA for something in such state of active development.

Why don't we have Alex tree go straight to -next like I do with Kumar for example ? That way I don't need to have his branch sit in my tree for weeks before I push it out to Linus.

Cheers,
Ben.

author
#16 lore raw
Quoted 15 lines · Benjamin Herrenschmidt
Quoted 7 lines · Avi Kivity
Quoted 6 lines · Paul Mackerras

I just noticed that the branch you asked Linus to pull includes none of the patches that Alex sent you in the last batch, in the email with subject "[PULL 00/56] ppc patch queue 2012-03-15" sent on March 15, where he asked you to pull git://github.com/agraf/linux-2.6.git
for-upstream.

Quoted 6 lines
Quoted 2 lines

What happened? Did they get lost in the re-signing, or is there some reason you thought they shouldn't go in?

That pull request was send three days before the merge window opened; patches are supposed to cook for a while in -next before being merged, especially large trees like that one.

Quoted 8 lines

These are all powerpc specific patches that have been cooking in Alex tree for a while and elsewhere before that. They almost only affect arch/powerpc/kvm, and as such don't really need a lot of integration testing in -next. A bit for sure but not necessarily monthes.

The current process is such that it takes absolutely forever for our patches to get in, which is a major PITA for something in such state of active development.

If the patches were posted two weeks earlier, they would have gone in.

Quoted 3 lines

Why don't we have Alex tree go straight to -next like I do with Kumar for example ? That way I don't need to have his branch sit in my tree for weeks before I push it out to Linus.

There isn't a lot of common kvm code, but what there is needs to be synchronized.

Signature · 1 line
error compiling committee.c: too many arguments to function
#17 lore raw
Quoted 5 lines · Avi Kivity
Quoted 3 lines

The current process is such that it takes absolutely forever for our patches to get in, which is a major PITA for something in such state of active development.

If the patches were posted two weeks earlier, they would have gone in.

I believe on our side they were, but Alex took a while to make up his tree ... oh well..

Quoted 6 lines
Quoted 3 lines

Why don't we have Alex tree go straight to -next like I do with Kumar for example ? That way I don't need to have his branch sit in my tree for weeks before I push it out to Linus.

There isn't a lot of common kvm code, but what there is needs to be synchronized.

At least we should be able to get the important bug fixes in now, I see that you agree in your reply to Paulus so that's good :-)

Cheers,
Ben.

#18 lore raw
Quoted 9 lines · Benjamin Herrenschmidt
Quoted 5 lines · Avi Kivity
Quoted 3 lines

The current process is such that it takes absolutely forever for our patches to get in, which is a major PITA for something in such state of active development.

If the patches were posted two weeks earlier, they would have gone in.

I believe on our side they were, but Alex took a while to make up his tree ... oh well..

Yes, because to me the kvm-ppc-next tree basically is the same semantically as -next for Avi. It's where patches cook a while to make sure they actually work. Nobody tests KVM PPC patches against kvm's master tree. All testing (compile and execution) happens against kvm-ppc-next.

That's why I don't see the point in having it cook again in Avi's tree. At the end of the day the patches will surely become way too chewy ;).

The alternative would be that I don't have a -next tree, just collect patches and immediately send them to Avi. That way the main kvm tree would be broken more often, but at least we don't get these horrible synchronization latencies.

Alex

author
#19 lore raw
Quoted 7 lines · Alexander Graf
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 4 lines · Avi Kivity
Quoted 3 lines

The current process is such that it takes absolutely forever for our patches to get in, which is a major PITA for something in such state of active development.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

If the patches were posted two weeks earlier, they would have gone in.

I believe on our side they were, but Alex took a while to make up his tree ... oh well..

Yes, because to me the kvm-ppc-next tree basically is the same semantically as -next for Avi. It's where patches cook a while to make sure they actually work. Nobody tests KVM PPC patches against kvm's master tree. All testing (compile and execution) happens against kvm-ppc-next.

That's why I don't see the point in having it cook again in Avi's tree. At the end of the day the patches will surely become way too chewy ;).

kvm.git next is exposed to linux-next, where they get tested quite a lot. Granted it's mostly build testing, and people are unlikely to test kvm there, but they will test the non-kvm bits that creep in there.

Quoted 1 line

The alternative would be that I don't have a -next tree, just collect patches and immediately send them to Avi. That way the main kvm tree would be broken more often, but at least we don't get these horrible synchronization latencies.

That works too. Don't post immediately; 2-3 week batches would reduce noise.

Signature · 1 line
error compiling committee.c: too many arguments to function
#20 lore raw
Quoted 17 lines · Avi Kivity
Quoted 7 lines · Alexander Graf
Quoted 5 lines · Benjamin Herrenschmidt
Quoted 4 lines · Avi Kivity
Quoted 3 lines

The current process is such that it takes absolutely forever for our patches to get in, which is a major PITA for something in such state of active development.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

If the patches were posted two weeks earlier, they would have gone in.

I believe on our side they were, but Alex took a while to make up his tree ... oh well..

Yes, because to me the kvm-ppc-next tree basically is the same semantically as -next for Avi. It's where patches cook a while to make sure they actually work. Nobody tests KVM PPC patches against kvm's master tree. All testing (compile and execution) happens against kvm-ppc-next.

That's why I don't see the point in having it cook again in Avi's tree. At the end of the day the patches will surely become way too chewy ;).

Quoted 8 lines

kvm.git next is exposed to linux-next, where they get tested quite a lot. Granted it's mostly build testing, and people are unlikely to test kvm there, but they will test the non-kvm bits that creep in there.

Quoted 1 line

The alternative would be that I don't have a -next tree, just collect patches and immediately send them to Avi. That way the main kvm tree would be broken more often, but at least we don't get these horrible synchronization latencies.

That works too. Don't post immediately; 2-3 week batches would reduce noise.

Ok, let's try to move to that scheme then :). I'll just send pull requests to you even though there are unprocessed other kvm ppc patches still around and we'll stage everything in your tree.

Alex

#21 lore raw
In reply to Avi Kivity
Quoted 6 lines · Avi Kivity

kvm.git next is exposed to linux-next, where they get tested quite a lot. Granted it's mostly build testing, and people are unlikely to test
kvm there, but they will test the non-kvm bits that creep in there.

Quoted 7 lines
Quoted 1 line

The alternative would be that I don't have a -next tree, just

collect patches and immediately send them to Avi. That way the main kvm tree would be broken more often, but at least we don't get these horrible synchronization latencies.

That works too. Don't post immediately; 2-3 week batches would reduce noise.

Or do like I do with Kumar for FSL stuff... his stuff gets pulled via my tree but his tree is in linux-next as well. There's no reason not to do that.

That way, his next branch gets linux-next coverage whether it's in my tree or not, and I pull when I put the final powerpc-next together, which gives me a chance to do a quick vet "just in case" and sort out any major conflict before it all goes to Linus.

Cheers,
Ben.

author
#22 lore raw
Quoted 15 lines · Benjamin Herrenschmidt
Quoted 6 lines · Avi Kivity

kvm.git next is exposed to linux-next, where they get tested quite a lot. Granted it's mostly build testing, and people are unlikely to test
kvm there, but they will test the non-kvm bits that creep in there.

Quoted 7 lines
Quoted 1 line

The alternative would be that I don't have a -next tree, just

collect patches and immediately send them to Avi. That way the main kvm tree would be broken more often, but at least we don't get these horrible synchronization latencies.

That works too. Don't post immediately; 2-3 week batches would reduce noise.

Quoted 9 lines

Or do like I do with Kumar for FSL stuff... his stuff gets pulled via my tree but his tree is in linux-next as well. There's no reason not to do that.

That way, his next branch gets linux-next coverage whether it's in my tree or not, and I pull when I put the final powerpc-next together, which gives me a chance to do a quick vet "just in case" and sort out any major conflict before it all goes to Linus.

Sure, that works too.

Signature · 1 line
error compiling committee.c: too many arguments to function
#23 lore raw
Quoted 18 lines · Avi Kivity
Quoted 17 lines · Benjamin Herrenschmidt
Quoted 5 lines · Avi Kivity

kvm.git next is exposed to linux-next, where they get tested quite a lot. Granted it's mostly build testing, and people are unlikely to test
kvm there, but they will test the non-kvm bits that creep in there.

Quoted 7 lines
Quoted 1 line

The alternative would be that I don't have a -next tree, just

collect patches and immediately send them to Avi. That way the main kvm tree would be broken more often, but at least we don't get these horrible synchronization latencies.

That works too. Don't post immediately; 2-3 week batches would reduce noise.

Or do like I do with Kumar for FSL stuff... his stuff gets pulled via my tree but his tree is in linux-next as well. There's no reason not to do that.

Quoted 6 lines
Quoted 5 lines

That way, his next branch gets linux-next coverage whether it's in my tree or not, and I pull when I put the final powerpc-next together, which gives me a chance to do a quick vet "just in case" and sort out any major conflict before it all goes to Linus.

Sure, that works too.

Sounds even easier to me. So how do I get my tree into linux-next?

Alex

author
#24 lore raw
Quoted 8 lines · Alexander Graf
Quoted 8 lines
Quoted 8 lines

tree but his tree is in linux-next as well. There's no reason not to do that.

That way, his next branch gets linux-next coverage whether it's in my tree or not, and I pull when I put the final powerpc-next together, which gives me a chance to do a quick vet "just in case" and sort out any major conflict before it all goes to Linus.

Quoted 4 lines
Quoted 1 line

Sure, that works too.

Or do like I do with Kumar for FSL stuff... his stuff gets pulled via my

Sounds even easier to me. So how do I get my tree into linux-next?

Send a note to Stephen Rothwell (copied).

Signature · 1 line
error compiling committee.c: too many arguments to function
#25 lore raw
In reply to Avi Kivity
Quoted 7 lines · Avi Kivity
Quoted 6 lines · Paul Mackerras

I just noticed that the branch you asked Linus to pull includes none of the patches that Alex sent you in the last batch, in the email with subject "[PULL 00/56] ppc patch queue 2012-03-15" sent on March 15, where he asked you to pull git://github.com/agraf/linux-2.6.git
for-upstream.

Quoted 6 lines
Quoted 2 lines

What happened? Did they get lost in the re-signing, or is there some reason you thought they shouldn't go in?

That pull request was send three days before the merge window opened; patches are supposed to cook for a while in -next before being merged, especially large trees like that one.

OK. Then there are about six commits from that string that are small, simple fixes for bugs introduced in earlier patches, that only affect arch/powerpc. They should go into Linus' tree as soon as possible.

How do you want to handle those? Do you want them as patches or as a git tree, and if the latter, what do you prefer that I base them on? Or should I send them in via Ben?

Paul.

author
#26 lore raw
Quoted 11 lines · Paul Mackerras
Quoted 10 lines · Avi Kivity
Quoted 8 lines · Paul Mackerras

I just noticed that the branch you asked Linus to pull includes none of the patches that Alex sent you in the last batch, in the email with subject "[PULL 00/56] ppc patch queue 2012-03-15" sent on March 15, where he asked you to pull git://github.com/agraf/linux-2.6.git
for-upstream.

What happened? Did they get lost in the re-signing, or is there some reason you thought they shouldn't go in?

Quoted 7 lines
Quoted 3 lines

That pull request was send three days before the merge window opened; patches are supposed to cook for a while in -next before being merged, especially large trees like that one.

OK. Then there are about six commits from that string that are small, simple fixes for bugs introduced in earlier patches, that only affect arch/powerpc. They should go into Linus' tree as soon as possible.

Sure.

Quoted 3 lines

How do you want to handle those? Do you want them as patches or as a git tree, and if the latter, what do you prefer that I base them on? Or should I send them in via Ben?

Git tree against upstream please.

Signature · 1 line
error compiling committee.c: too many arguments to function