Received: from eggs.gnu.org ([2001:4830:134:3::10]:59405) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VrYmG-0007LT-Ji for qemu-devel@nongnu.org; Fri, 13 Dec 2013 14:52:04 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1VrYmC-00074O-B7 for qemu-devel@nongnu.org; Fri, 13 Dec 2013 14:52:00 -0500 Message-ID: <1386962330.10013.302.camel@snotra.buserror.net> From: Scott Wood Date: Fri, 13 Dec 2013 13:18:50 -0600 In-Reply-To: References: <1386768216-33686-1-git-send-email-agraf@suse.de> Content-Type: text/plain; charset="UTF-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH] roms: Flush icache when writing roms to guest memory List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Peter Maydell Cc: "Bogdan.Vlad@freescale.com" , Alexander Graf , QEMU Developers , Marc Zyngier , "qemu-ppc@nongnu.org" , "Varun.Sethi@freescale.com" , "mihai.caraman@freescale.com" On Wed, 2013-12-11 at 13:56 +0000, Peter Maydell wrote: > On 11 December 2013 13:23, Alexander Graf wrote: > > The guest expects that its data and instruction cache view of the world > > is 100% consistent when it initially boots. This works just fine on > > initial rom population for the first boot. > > > > However, when we reboot and then repopulate the rom region there could > > be old code still stuck in the instruction cache, giving the guest an > > inconsistent view of the world when we're using kvm. > > > > So we need to invalidate the icache every time we write a rom into guest > > address space. We do not need to do this for every DMA since the guest > > expects it has to flush the icache manually in that case. > > > @@ -2033,6 +2034,13 @@ void cpu_physical_memory_write_rom(hwaddr addr, > > ptr = qemu_get_ram_ptr(addr1); > > memcpy(ptr, buf, l); > > invalidate_and_set_dirty(addr1, l); > > + if (kvm_enabled()) { > > + /* > > + * The guest may want to directly execute from the rom region, > > + * so we better invalidate its icache > > + */ > > + flush_icache_range((uintptr_t)ptr, (uintptr_t)ptr + l); > > + } > > I bet these aren't the only places where code gets written > to guest memory. Also are you sure flush_icache_range() > works correctly when multiple threads (multiple vCPUs, > potentially executing on different host CPUs) are involved? On PPC these cache operations broadcast, and are the architecturally defined way of doing self-modifying code. > The TCG case only needs to care about "this thread writes code > to memory that it will itself later execute", not any kind of > cross-host-CPU flushing. Can't the TCG thread get migrated between CPUs? > There was a huge thread on kvmarm earlier this year > https://lists.cs.columbia.edu/pipermail/kvmarm/2013-August/006716.html > about a similar sort of issue, and I think the conclusion was that > the kernel basically had to deal with the problem itself [though > the thread is rather confusing...]. I've cc'd Marc Z in the hope > he remembers the ARM specific detail... Hmm, a good point is raised in that thread regarding what happens if a page is swapped out and then back in: https://lists.cs.columbia.edu/pipermail/kvmarm/2013-August/006738.html I think the usual mechanism PPC booke uses to handle this is currently not effective with KVM because we get the pages via __get_user_pages fast() rather than by enabling execute permission in an ISI (see the second instance of set_pte_filter() in arch/powerpc/mm/pgtable.c). Even if we fix that to invoke the cache cleaning code when KVM acquires a page, though, QEMU would still need to flush if it modifies/loads code on a page that may already be marked in the kernel as having been cleaned. -Scott