Received: by 2002:a5d:51d0:0:0:0:0:0 with SMTP id n16csp5127713wrv; Tue, 6 Aug 2019 04:20:18 -0700 (PDT) X-Google-Smtp-Source: APXvYqw2zliVJSl4xkT+cBYLfYkBjTiWOeqXWsKgYBzIE0I70tfnKxNZRkrplp3Tcy0kSFMoskrz X-Received: by 2002:a37:8a81:: with SMTP id m123mr2607779qkd.360.1565090417918; Tue, 06 Aug 2019 04:20:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1565090417; cv=none; d=google.com; s=arc-20160816; b=OcC/yY5Cei+RKzIivk+PRmD6sns+17Awfo3+uJ6hYJhCue2z/pRdakFp8JI9KMNCqj gvwWCPw8XVX0MYpI+RhxACoxCZ9hl7HLgikzy2lsb+tnh0QemnttNiZQ7GvjJimJx97c iFo/LsBLgedAcdsAgH/veO+xzZBb/ooIOqfMbhQKbEmJnDBsEkmbZeLnfe1obk5XMDga NPlNSpuexP7x+GWan0oqG5sTjubm3HtRNfy20aMo3PvcpU3ch9MugYBLn4vQ8LjeXc1N 8273gcqRBWV1+5hsSASFOQXNVQ34YVaNuTwbJvzN2QiYwZQrylZJoMHQ9H9ZkHd99fcV jvdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:to:from:date; bh=NVHyr1XlMMfUFugD8BC/lepAJXD/lIZIJDOsSXQfvAA=; b=OolCSo8kgPGypd2n3kvFA6NTqY+5bA02Bj8MX2OFLcl8hwacvaRc5Y4toFjoA9F8Cx sujpdFPxBRfiEfKz1YA2I5KukeybCWDkQK3q3fMADWC7V+Pgey1HlMdGVNBv6BQmass4 iDJzY8uPjCtAB9Ib3M47czM0DXbec/DAiUldhCrYKEMykVpkrOTDjwksx6+3dDX32i0a DyDGODXvdMWRFiev/KL7f0qj4HkmoidsfaAf3Ady0Jxiwqced0mOLwzoYlUKLp8RLBup EexV/XQCx3wf3WPLeCP0+CkYX9G6qbninq3uJQeB2tl0yLmG/Z8gHOB9j9NYIGjud5qc giHA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org" Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id x202si47863029qka.295.2019.08.06.04.20.17 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 06 Aug 2019 04:20:17 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; spf=pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org" Received: from localhost ([::1]:60342 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1huxVp-0005GT-Gt for alex.bennee@linaro.org; Tue, 06 Aug 2019 07:20:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52140) by lists.gnu.org with esmtp (Exim 4.86_2) (envelope-from ) id 1huxVj-0005FA-Oi for qemu-arm@nongnu.org; Tue, 06 Aug 2019 07:20:13 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1huxVh-0007XS-US for qemu-arm@nongnu.org; Tue, 06 Aug 2019 07:20:11 -0400 Received: from szxga07-in.huawei.com ([45.249.212.35]:54192 helo=huawei.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1huxVh-0007T2-C0; Tue, 06 Aug 2019 07:20:09 -0400 Received: from DGGEMS401-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id A08A8FDEB713B425BF00; Tue, 6 Aug 2019 19:19:55 +0800 (CST) Received: from localhost (10.202.226.61) by DGGEMS401-HUB.china.huawei.com (10.3.19.201) with Microsoft SMTP Server id 14.3.439.0; Tue, 6 Aug 2019 19:19:46 +0800 Date: Tue, 6 Aug 2019 12:19:37 +0100 From: Jonathan Cameron To: QEMU Developers Message-ID: <20190806121937.00000c4a@huawei.com> In-Reply-To: <20190625112752.83188-1-Jonathan.Cameron@huawei.com> References: <20190625112752.83188-1-Jonathan.Cameron@huawei.com> Organization: Huawei X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; i686-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable X-Originating-IP: [10.202.226.61] X-CFilter-Loop: Reflected X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 45.249.212.35 Subject: Re: [Qemu-arm] [RFC PATCH 0/7] qemu: CCIX pcie config space emulation X-BeenThere: qemu-arm@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , qemu-arm , linuxarm@huawei.com, jcm@redhat.com, Auger Eric Errors-To: qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org Sender: "Qemu-arm" X-TUID: EUC1yDSs4n0e For reference alongside this patch set. Evaluation version of the CCIX 1.0a base specification now available, (though there is a form to complete and license agreement).. https://www.ccixconsortium.com/ccix-library/download-form/ Thanks, Jonathan On Tue, 25 Jun 2019 19:27:45 +0800 Jonathan Cameron wrote: > CCIX topologies are 'layered' on top of PCIe tree topologies. > This is done primarily by allowing a single CCIX device to appear as > multiple disjoint nodes in the PCIe tree. >=20 > This layering is described via extensive PCIe DVSEC extended > capabilities in PCIe config space across all the functions that > are present in the device (some placement rules apply). >=20 > The extremely flexible nature of allowed topologies makes the > development of generic firmware and OS software difficult if we rely > on actual hardware setups, due to the large test set that is necessary. >=20 > To enable the ongoing work on EDK2 and within the Linux kernel and > userspace, this series provides the bare bones of what is necessary > to be able to test 'configuration' of a CCIX topology. Note > that no actual CCIX data flow is being emulated within this patchset, > merely a substantial subset of the interface that allows the topologies > to be configured. Testing has to rely on verification based on > the result rather than true emulation of the coherency protocol. > (that is a very different form of emulation for which other tools > are perhaps better suited). >=20 > For information on how to do the coherency protocol configuration, > see the forthcoming CCIX SW guide, in conjunction with the CCIX 1.0 > Base Specification. >=20 > An example of a 2x2 mesh with a spur to the host can be run with: >=20 > qemu-system-aarch64 -M virt -m 1024 -cpu cortex-a53 \ > ... > -device ioh3420,id=3Droot_port1 \ > -device ccix-upstream-port,num_links=3D4,primaryport=3Dtrue,rsam_entries= =3D4,ccix_device=3D"ccix_dev1",bus=3Droot_port1,addr=3D0.0,multifunction=3D= "on",id=3Dup0,port_id=3D0 \ > -device ccix-downstream-port,ccix_device=3D"ccix_dev1",bus=3Dup0,slot=3D= 0,chassis=3D2,id=3Dbus_top,port_id=3D1 \ > -device ccix-downstream-port,ccix_device=3D"ccix_dev1",bus=3Dup0,slot=3D= 1,chassis=3D2,id=3Dbus_left,port_id=3D2 \ > -device ccix-ep,primaryport=3Dfalse,home_agents=3D1,request_agents=3D1,c= cix_device=3D"ccix_dev1",bus=3Droot_port1,addr=3D0.1,multifunction=3D"on" \ > -device ccix-upstream-port,num_links=3D4,primaryport=3Dtrue,rsam_entries= =3D4,ccix_device=3D"ccix_dev2",bus=3Dbus_top,addr=3D0.0,multifunction=3D"on= ",id=3Dup1,port_id=3D0 \ > -device ccix-downstream-port,ccix_device=3D"ccix_dev2",bus=3Dup1,slot=3D= 0,chassis=3D3,id=3Dbus_right,port_id=3D1 \ > -device ccix-ep,primaryport=3Dfalse,request_agents=3D2,ccix_device=3D"cc= ix_dev2",bus=3Dbus_top,addr=3D0.1,multifunction=3D"on" \ > -device ccix-upstream-port,num_links=3D4,primaryport=3Dtrue,rsam_entries= =3D4,ccix_device=3D"ccix_dev3",bus=3Dbus_left,addr=3D0.0,multifunction=3D"o= n",id=3Dup2,port_id=3D0 \ > -device ccix-downstream-port,ccix_device=3D"ccix_dev3",bus=3Dup2,slot=3D= 0,chassis=3D4,id=3Dbus_bottom,port_id=3D1,multifunciton=3D"on" \ > -device ccix-ep,primaryport=3Dfalse,request_agents=3D2,ccix_device=3D"cc= ix_dev3",bus=3Dbus_left,addr=3D0.1,multifunction=3D"on" \ > -device ccix-ep,primaryport=3Dtrue,request_agents=3D2,ccix_device=3D"cci= x_dev4",bus=3Dbus_right,addr=3D0.0,port_id=3D0 \ > -device ccix-ep,primaryport=3Dtrue,request_agents=3D2,ccix_device=3D"cci= x_dev4",bus=3Dbus_bottom,addr=3D0.0,port_id=3D1 >=20 >=20 > I'm not going to try drawing all the detail (it's bad enough > trying to draw these in inkscape, but in a very much simplifed > fashion, this generates. >=20 > ----------------- > | Host | > | |=20 > --- root_port1-- > | > | > v > ----------------- --------------- > | ccix_dev1 | -------> | ccix_dev2 | > ----------------- --------------- > | | > V V > ----------------- --------------- > | ccix_dev3 | -------> | ccix_dev4 | > ----------------- --------------- >=20 > $lspci -t > -[0000:00]-+-00.0 > +-01.0-[01-08]--+-00.0-[02-08]--+-00.0-[03-05]--+-00.0-[04-05]= ----00.0-[05]----00.0 > | | | = \-00.1 > | | \-01.0-[06-08]= --+-00.0-[07-08]----00.0-[08]----00.0 > | | = \-00.1 > | \-00.1 > =20 > RFC questions: >=20 > 1. The nature of CCIX devices is that we need to extend normal > PCI devices, slots, and ports. I could not find an elegant way of > doing this without lots of code replication. The current solution > just exposes some internal functions from xio3130 port implementation= s. > Is there a better way to do this? >=20 > 2. The association of the different PCIDevices to a given CCIX device is > currently somewhat of a hack. Can anyone suggest a cleaner solution > for this? I can improved the current implementation, but don't really > like that we basically search for all the parts whenever a cross > device implementation is needed. >=20 > 3. Is emulation via a large number of PCIe devices like this a good > approach or is there a nicer way to handle it? Given we need to > be able to 'spread' the CCIX device configuration across multiple > PCIe functions anyway perhaps this is the most sensible approach. >=20 > 4. I've cut and paste a 100+ lines of code from each of the xio3130_* > modules as we are also implemening PCIE up and downstream ports > and as this is a emulation only device, we might as well use the > same register set. There are various possible ways to avoid this: > * Add a library with the shared code in it. > * Add an xio3130_upstream.h header and similar to allow the CCIX > port modules to call those functions directly. > * Don't worry about the replication in the interests of keeping > the code structure simple. >=20 > 5. I'm not that familiar with qemu 'style' yet, so pointers on structures > to use etc most welcome. >=20 > Note that not all elements of CCIX are supported by the current implement= ation, > for example slave agents and error records are missing. These will follow > either in later revisions or as follow patches. We also have no actual > accelerator functions in the current design and hence no mapping to RAs. >=20 > Only a subset of configuration constraints are currently implemented. > This will want tightenning up in future. >=20 > As we don't have any actual chunks of the spec in here so I haven't > added the statement from the trademark grant that follows. >=20 > Thanks, >=20 > Jonathan >=20 > This patch is being distributed by the CCIX Consortium, Inc. (CCIX) to > you and other parties that are paticipating (the "participants") in > qemu with the understanding that the participants will use CCIX's > name and trademark only when this patch is used in association with > qemu. >=20 > CCIX is also distributing this patch to these participants with the > understanding that if any portion of the CCIX specification will be > used or referenced in qemu, the participants will not modify the cited > portion of the CCIX specification and will give CCIX propery copyright > attribution by including the following copyright notice with > the cited part of the CCIX specification: > "=A9 2019 CCIX CONSORTIUM, INC. ALL RIGHTS RESERVED." >=20 > Jonathan Cameron (7): > Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy. > pci: Add Huawei vendor ID and Huawei Emulated CCIX Device IDs. > pci: CCIX config space emulation library. > pci-bridge: CCIX capable PCIE/CCIX switch upstream port. > pci-bridge: CCIX capable PCIE/CCIX switch downstream port > misc: CCIX endpoint function > Temp: Add to ARM64 makefiles for testing >=20 > default-configs/arm-softmmu.mak | 3 +- > hw/misc/Kconfig | 5 + > hw/misc/Makefile.objs | 1 + > hw/misc/ccix-ep.c | 112 ++ > hw/pci-bridge/Kconfig | 5 + > hw/pci-bridge/Makefile.objs | 1 + > hw/pci-bridge/ccix_downstream.c | 222 ++++ > hw/pci-bridge/ccix_upstream.c | 197 ++++ > hw/pci/Kconfig | 3 + > hw/pci/Makefile.objs | 1 + > hw/pci/ccix_lib.c | 1299 +++++++++++++++++++++ > include/hw/misc/ccix.h | 28 + > include/hw/pci/pci_ids.h | 7 + > include/standard-headers/linux/pci_regs.h | 3 +- > 14 files changed, 1885 insertions(+), 2 deletions(-) > create mode 100644 hw/misc/ccix-ep.c > create mode 100644 hw/pci-bridge/ccix_downstream.c > create mode 100644 hw/pci-bridge/ccix_upstream.c > create mode 100644 hw/pci/ccix_lib.c > create mode 100644 include/hw/misc/ccix.h >=20