Xilinx Alveo/XRT patch overview

From: Sonal Santan <[email protected]>

Describe Alveo XRT driver architecture and provide basic overview of Xilinx Alveo platform.

Signed-off-by Sonal Santan <[email protected]>

---
 Documentation/fpga/index.rst |   1 +
 Documentation/fpga/xrt.rst   | 588 +++++++++++++++++++++++++++++++++++
 2 files changed, 589 insertions(+)
 create mode 100644 Documentation/fpga/xrt.rst

Documentation/fpga/index.rst +1 -0

index f80f95667ca2..30134357b70d 100644
@@ -8,6 +8,7 @@ fpga
     :maxdepth: 1
 
     dfl
+    xrt
 
 .. only::  subproject and html
 

Documentation/fpga/xrt.rst new file +3 -0

new file mode 100644
index 000000000000..9f37d46459b0
@@ -0,1 +1,588 @@
+==================================
+XRTV2 Linux Kernel Driver Overview
+==================================

Documentation/fpga/xrt.rst new file +31 -0

+provides basic infrastructure for the Alveo platform like PCIe connectivity, board management,
+Dynamic Function Exchange (DFX), sensors, clocking, reset, and security. User partition contains
+user compiled binary which is loaded by a process called DFX also known as partial reconfiguration.
+
+Physical partitions require strict HW compatibility with each other for DFX to work properly.
+Every physical partition has two interface UUIDs: *parent* UUID and *child* UUID. For simple
+single stage platforms Shell → User forms parent child relationship. For complex two stage
+platforms Base → Shell → User forms the parent child relationship chain.
+
+.. note::
+   Partition compatibility matching is key design component of Alveo platforms and XRT. Partitions
+   have child and parent relationship. A loaded partition exposes child partition UUID to advertise
+   its compatibility requirement for child partition. When loading a child partition the xmgmt
+   management driver matches parent UUID of the child partition against child UUID exported by the
+   parent. Parent and child partition UUIDs are stored in the *xclbin* (for user) or *xsabin* (for
+   base and shell). Except for root UUID, VSEC, hardware itself does not know about UUIDs. UUIDs are
+   stored in xsabin and xclbin.
+
+
+The physical partitions and their loading is illustrated below::
+
+            SHELL                               USER
+        +-----------+                  +-------------------+
+        |           |                  |                   |
+        | VSEC UUID | CHILD     PARENT |    LOGIC UUID     |
+        |           o------->|<--------o                   |
+        |           | UUID       UUID  |                   |
+        +-----+-----+                  +--------+----------+
+              |                                 |
+              .                                 .
+              |                                 |

Documentation/fpga/xrt.rst new file +74 -0

+5. Program the clocks (ClockWiz) driving the User partition
+6. Wait for memory controller (MIG) calibration
+
+`Platform Loading Overview <https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`_
+provides more detailed information on platform loading.
+
+xsabin
+------
+
+Each Alveo platform comes packaged with its own xsabin. The xsabin is trusted component of the
+platform. For format details refer to :ref:`xsabin/xclbin Container Format`. xsabin contains
+basic information like UUIDs, platform name and metadata in the form of device tree. See
+:ref:`Device Tree Usage` for details and example.
+
+xclbin
+------
+
+xclbin is compiled by end user using
+`Vitis <https://www.xilinx.com/products/design-tools/vitis/vitis-platform.html>`_ tool set from
+Xilinx. The xclbin contains sections describing user compiled acceleration engines/kernels, memory
+subsystems, clocking information etc. It also contains bitstream for the user partition, UUIDs,
+platform name, etc. xclbin uses the same container format as xsabin which is described below.
+
+
+xsabin/xclbin Container Format
+------------------------------
+
+xclbin/xsabin is ELF-like binary container format. It is structured as series of sections.
+There is a file header followed by several section headers which is followed by sections.
+A section header points to an actual section. There is an optional signature at the end.
+The format is defined by header file ``xclbin.h``. The following figure illustrates a
+typical xclbin::
+
+
+          +---------------------+
+          |                     |
+          |       HEADER        |
+          +---------------------+
+          |   SECTION  HEADER   |
+          |                     |
+          +---------------------+
+          |         ...         |
+          |                     |
+          +---------------------+
+          |   SECTION  HEADER   |
+          |                     |
+          +---------------------+
+          |       SECTION       |
+          |                     |
+          +---------------------+
+          |         ...         |
+          |                     |
+          +---------------------+
+          |       SECTION       |
+          |                     |
+          +---------------------+
+          |      SIGNATURE      |
+          |      (OPTIONAL)     |
+          +---------------------+
+
+
+xclbin/xsabin files can be packaged, un-packaged and inspected using XRT utility called
+**xclbinutil**. xclbinutil is part of XRT open source software stack. The source code for
+xclbinutil can be found at https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbinutil
+
+For example to enumerate the contents of a xclbin/xsabin use the *--info* switch as shown
+below::
+
+  xclbinutil --info --input /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
+  xclbinutil --info --input /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsabin
+
+
+Device Tree Usage
+-----------------

Documentation/fpga/xrt.rst new file +309 -0

+
+  /dts-v1/;
+
+  /{
+	logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
+
+	schema_version {
+		major = <0x01>;
+		minor = <0x00>;
+	};
+
+	interfaces {
+
+		@0 {
+			interface_uuid = "862c7020a250293e32036f19956669e5";
+		};
+	};
+
+	addressable_endpoints {
+
+		ep_blp_rom_00 {
+			reg = <0x00 0x1f04000 0x00 0x1000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+		};
+
+		ep_card_flash_program_00 {
+			reg = <0x00 0x1f06000 0x00 0x1000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_quad_spi-1.0\0axi_quad_spi";
+			interrupts = <0x03 0x03>;
+		};
+
+		ep_cmc_firmware_mem_00 {
+			reg = <0x00 0x1e20000 0x00 0x20000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+			firmware {
+				firmware_product_name = "cmc";
+				firmware_branch_name = "u50";
+				firmware_version_major = <0x01>;
+				firmware_version_minor = <0x00>;
+			};
+		};
+
+		ep_cmc_intc_00 {
+			reg = <0x00 0x1e03000 0x00 0x1000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
+			interrupts = <0x04 0x04>;
+		};
+
+		ep_cmc_mutex_00 {
+			reg = <0x00 0x1e02000 0x00 0x1000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+		};
+
+		ep_cmc_regmap_00 {
+			reg = <0x00 0x1e08000 0x00 0x2000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+			firmware {
+				firmware_product_name = "sc-fw";
+				firmware_branch_name = "u50";
+				firmware_version_major = <0x05>;
+			};
+		};
+
+		ep_cmc_reset_00 {
+			reg = <0x00 0x1e01000 0x00 0x1000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+		};
+
+		ep_ddr_mem_calib_00 {
+			reg = <0x00 0x63000 0x00 0x1000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+		};
+
+		ep_debug_bscan_mgmt_00 {
+			reg = <0x00 0x1e90000 0x00 0x10000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-debug_bridge-1.0\0debug_bridge";
+		};
+
+		ep_ert_base_address_00 {
+			reg = <0x00 0x21000 0x00 0x1000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+		};
+
+		ep_ert_command_queue_mgmt_00 {
+			reg = <0x00 0x40000 0x00 0x10000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
+		};
+
+		ep_ert_command_queue_user_00 {
+			reg = <0x00 0x40000 0x00 0x10000>;
+			pcie_physical_function = <0x01>;
+			compatible = "xilinx.com,reg_abs-ert_command_queue-1.0\0ert_command_queue";
+		};
+
+		ep_ert_firmware_mem_00 {
+			reg = <0x00 0x30000 0x00 0x8000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+			firmware {
+				firmware_product_name = "ert";
+				firmware_branch_name = "v20";
+				firmware_version_major = <0x01>;
+			};
+		};
+
+		ep_ert_intc_00 {
+			reg = <0x00 0x23000 0x00 0x1000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
+			interrupts = <0x05 0x05>;
+		};
+
+		ep_ert_reset_00 {
+			reg = <0x00 0x22000 0x00 0x1000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+		};
+
+		ep_ert_sched_00 {
+			reg = <0x00 0x50000 0x00 0x1000>;
+			pcie_physical_function = <0x01>;
+			compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
+			interrupts = <0x09 0x0c>;
+		};
+
+		ep_fpga_configuration_00 {
+			reg = <0x00 0x1e88000 0x00 0x8000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
+			interrupts = <0x02 0x02>;
+		};
+
+		ep_icap_reset_00 {
+			reg = <0x00 0x1f07000 0x00 0x1000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+		};
+
+		ep_mailbox_mgmt_00 {
+			reg = <0x00 0x1f10000 0x00 0x10000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-mailbox-1.0\0mailbox";
+			interrupts = <0x00 0x00>;
+		};
+
+		ep_mailbox_user_00 {
+			reg = <0x00 0x1f00000 0x00 0x10000>;
+			pcie_physical_function = <0x01>;
+			compatible = "xilinx.com,reg_abs-mailbox-1.0\0mailbox";
+			interrupts = <0x08 0x08>;
+		};
+
+		ep_msix_00 {
+			reg = <0x00 0x00 0x00 0x20000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
+			pcie_bar_mapping = <0x02>;
+		};
+
+		ep_pcie_link_mon_00 {
+			reg = <0x00 0x1f05000 0x00 0x1000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+		};
+
+		ep_pr_isolate_plp_00 {
+			reg = <0x00 0x1f01000 0x00 0x1000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+		};
+
+		ep_pr_isolate_ulp_00 {
+			reg = <0x00 0x1000 0x00 0x1000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+		};
+
+		ep_uuid_rom_00 {
+			reg = <0x00 0x64000 0x00 0x1000>;
+			pcie_physical_function = <0x00>;
+			compatible = "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+		};
+
+		ep_xdma_00 {
+			reg = <0x00 0x00 0x00 0x10000>;
+			pcie_physical_function = <0x01>;
+			compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
+			pcie_bar_mapping = <0x02>;
+		};
+	};
+
+  }
+
+
+
+Deployment Models
+=================
+
+Baremetal
+---------
+
+In bare-metal deployments both MPF and UPF are visible and accessible. xmgmt driver binds to
+MPF. xmgmt driver operations are privileged and available to system administrator. The full
+stack is illustrated below::
+
+
+                            HOST
+
+                 [XMGMT]            [XUSER]
+                    |                  |
+                    |                  |
+                 +-----+            +-----+
+                 | MPF |            | UPF |
+                 |     |            |     |
+                 | PF0 |            | PF1 |
+                 +--+--+            +--+--+
+          ......... ^................. ^..........
+                    |                  |
+                    |   PCIe DEVICE    |
+                    |                  |
+                 +--+------------------+--+
+                 |         SHELL          |
+                 |                        |
+                 +------------------------+
+                 |         USER           |
+                 |                        |
+                 |                        |
+                 |                        |
+                 |                        |
+                 +------------------------+
+
+
+
+Virtualized
+-----------
+
+In virtualized deployments privileged MPF is assigned to host but unprivileged UPF
+is assigned to guest VM via PCIe pass-through. xmgmt driver in host binds to MPF.
+xmgmt driver operations are privileged and only accessible by hosting service provider.
+The full stack is illustrated below::
+
+
+                                 .............
+                  HOST           .    VM     .
+                                 .           .
+                 [XMGMT]         .  [XUSER]  .
+                    |            .     |     .
+                    |            .     |     .
+                 +-----+         .  +-----+  .
+                 | MPF |         .  | UPF |  .
+                 |     |         .  |     |  .
+                 | PF0 |         .  | PF1 |  .
+                 +--+--+         .  +--+--+  .
+          ......... ^................. ^..........
+                    |                  |
+                    |   PCIe DEVICE    |
+                    |                  |
+                 +--+------------------+--+
+                 |         SHELL          |
+                 |                        |
+                 +------------------------+
+                 |         USER           |
+                 |                        |
+                 |                        |
+                 |                        |
+                 |                        |
+                 +------------------------+
+
+
+
+Driver Modules
+==============
+
+xrt-lib.ko
+----------
+
+Repository of all subsystem drivers and pure software modules that can potentially
+be shared between xmgmt and xuser. All these drivers are structured as Linux
+*platform driver* and are instantiated by xmgmt (or xuser in future) based on meta
+data associated with hardware. The metadata is in the form of device tree as
+explained before.
+
+xmgmt.ko
+--------
+
+The xmgmt driver is a PCIe device driver driving MPF found on Xilinx's Alveo
+PCIE device. It consists of one *root* driver, one or more *partition* drivers
+and one or more *leaf* drivers. The root and MPF specific leaf drivers are in
+xmgmt.ko. The partition driver and other leaf drivers are in xrt-lib.ko.
+
+The instantiation of specific partition driver or leaf driver is completely data
+driven based on meta data (mostly in device tree format) found through VSEC
+capability and inside firmware files, such as xsabin or xclbin file. The root
+driver manages life cycle of multiple partition drivers, which, in turn, manages
+multiple leaf drivers. This allows a single set of driver code to support all

Documentation/fpga/xrt.rst new file +90 -0

+infrastructure of the MPF driver and resides in xmgmt.ko. This driver
+
+* manages one or more partition drivers
+* provides access to functionalities that requires pci_dev, such as PCIE config
+  space access, to other leaf drivers through parent calls
+* together with partition driver, facilities event callbacks for other leaf drivers
+* together with partition driver, facilities inter-leaf driver calls for other leaf
+  drivers
+
+When root driver starts, it will explicitly create an initial partition instance,
+which contains leaf drivers that will trigger the creation of other partition
+instances. The root driver will wait for all partitions and leaves to be created
+before it returns from it's probe routine and claim success of the initialization
+of the entire xmgmt driver.
+
+partition
+^^^^^^^^^
+
+The partition driver is a platform device driver whose life cycle is managed by
+root and does not have real IO mem or IRQ resources. It's part of the
+infrastructure of the MPF driver and resides in xrt-lib.ko. This driver
+
+* manages one or more leaf drivers so that multiple leaves can be managed as a group
+* provides access to root from leaves, so that parent calls, event notifications
+  and inter-leaf calls can happen
+
+In xmgmt, an initial partition driver instance will be created by root, which
+contains leaves that will trigger partition instances to be created to manage
+groups of leaves found on different partitions on hardware, such as VSEC, Shell,
+and User.
+
+leaves
+^^^^^^
+
+The leaf driver is a platform device driver whose life cycle is managed by
+a partition driver and may or may not have real IO mem or IRQ resources. They
+are the real meat of xmgmt and contains platform specific code to Shell and User
+found on a MPF.
+
+A leaf driver may not have real hardware resources when it merely acts as a driver
+that manages certain in-memory states for xmgmt. These in-memory states could be
+shared by multiple other leaves.
+
+Leaf drivers assigned to specific hardware resources drive specific subsystem in
+the device. To manipulate the subsystem or carry out a task, a leaf driver may ask
+help from root via parent calls and/or from other leaves via inter-leaf calls.
+
+A leaf can also broadcast events through infrastructure code for other leaves
+to process. It can also receive event notification from infrastructure about certain
+events, such as post-creation or pre-exit of a particular leaf.
+
+
+Driver Interfaces
+=================
+
+xmgmt Driver Ioctls
+-------------------
+
+Ioctls exposed by xmgmt driver to user space are enumerated in the following table:
+
+== ===================== ============================= ===========================
+#  Functionality         ioctl request code            data format
+== ===================== ============================= ===========================
+1  FPGA image download   XMGMT_IOCICAPDOWNLOAD_AXLF    xmgmt_ioc_bitstream_axlf
+2  CL frequency scaling  XMGMT_IOCFREQSCALE            xmgmt_ioc_freqscaling
+== ===================== ============================= ===========================
+
+xmgmt Driver Sysfs
+------------------
+
+xmgmt driver exposes a rich set of sysfs interfaces. Subsystem platform drivers
+export sysfs node for every platform instance.
+
+Every partition also exports its UUIDs. See below for examples::
+
+  /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
+  /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
+
+
+hwmon
+-----
+
+xmgmt driver exposes standard hwmon interface to report voltage, current, temperature,
+power, etc. These can easily be viewed using *sensors* command line utility.
+
+
+mailbox
+-------
+
+xmgmt communicates with user physical function driver via HW mailbox. Mailbox opcodes

Documentation/fpga/xrt.rst new file +7 -1

+
+
+Platform Security Considerations
+================================
+
+`Security of Alveo Platform <https://xilinx.github.io/XRT/master/html/security.html>`_
+discusses the deployment options and security implications in great detail.
--

2.17.1

-----Original Message-----
From: Moritz Fischer <[email protected]>
Sent: Monday, November 30, 2020 20:55
To: Sonal Santan <[email protected]>
Cc: [email protected]; [email protected]; Max Zhen <[email protected]>; Lizhi Hou <[email protected]>; Michal Simek <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers

Quoted 9 lines · Sonal Santan

From: Sonal Santan <[email protected]>

Describe Alveo XRT driver architecture and provide basic overview of Xilinx Alveo platform.

Signed-off-by Sonal Santan <[email protected]>

---
 Documentation/fpga/index.rst |   1 +
 Documentation/fpga/xrt.rst   | 588

+++++++++++++++++++++++++++++++++++

Quoted 5 lines

 2 files changed, 589 insertions(+)
 create mode 100644 Documentation/fpga/xrt.rst

(quoted diff) +0 -0

b/Documentation/fpga/index.rst index f80f95667ca2..30134357b70d

100644

Quoted 15 lines

Documentation/fpga/index.rst +1 -0

@@ -8,6 +8,7 @@ fpga
     :maxdepth: 1
 
     dfl
+    xrt
 
 .. only::  subproject and html
 

Documentation/fpga/xrt.rst new file +0 -0

new file mode 100644 index 000000000000..9f37d46459b0

Documentation/fpga/xrt.rst line 7

Quoted 7 lines

Documentation/fpga/xrt.rst new file +7 -0

@@ -0,1 +1,588 @@
+==================================
+XRTV2 Linux Kernel Driver Overview
+==================================
+
+XRTV2 drivers are second generation `XRT
+<https://github.com/Xilinx/XRT>`_ drivers which support `Alveo
+<https://www.xilinx.com/products/boards-and-kits/alveo.html>`_ PCIe

platforms from Xilinx.

Documentation/fpga/xrt.rst line 10

Quoted 3 lines

Documentation/fpga/xrt.rst new file +3 -0

+
+XRTV2 drivers support *subsystem* style data driven platforms where
+driver's configuration and behavior is determined by meta data provided

by platform (in *device tree* format).

Documentation/fpga/xrt.rst line 13

Quoted 3 lines

Documentation/fpga/xrt.rst new file +3 -0

+Primary management physical function (MPF) driver is called
+**xmgmt**. Primary user physical function (UPF) driver is called
+**xuser** and HW subsystem drivers are packaged into a library module

called **xrt-lib**, which is shared by **xmgmt** and **xuser** (WIP). WIP?

Documentation/fpga/xrt.rst

Quoted 6 lines

Documentation/fpga/xrt.rst new file +6 -0

+
+Alveo Platform Overview
+=======================
+
+Alveo platforms are architected as two physical FPGA partitions:
+*Shell* and *User*. Shell

Nit: The Shell provides ...

Documentation/fpga/xrt.rst line 22

Quoted 3 lines

Documentation/fpga/xrt.rst new file +3 -0

+provides basic infrastructure for the Alveo platform like PCIe
+connectivity, board management, Dynamic Function Exchange (DFX),
+sensors, clocking, reset, and security. User partition contains user

compiled binary which is loaded by a process called DFX also known as partial reconfiguration.

Documentation/fpga/xrt.rst line 24

Quoted 2 lines

Documentation/fpga/xrt.rst new file +2 -0

+
+Physical partitions require strict HW compatibility with each other for DFX

to work properly.

Documentation/fpga/xrt.rst line 27

Quoted 3 lines

Documentation/fpga/xrt.rst new file +3 -0

+Every physical partition has two interface UUIDs: *parent* UUID and
+*child* UUID. For simple single stage platforms Shell → User forms
+parent child relationship. For complex two stage platforms Base → Shell

→ User forms the parent child relationship chain.

Documentation/fpga/xrt.rst

Quoted 3 lines

Documentation/fpga/xrt.rst new file +3 -0

+
+.. note::
+   Partition compatibility matching is key design component of Alveo

platforms and XRT. Partitions

Documentation/fpga/xrt.rst line 31

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+   have child and parent relationship. A loaded partition exposes child

partition UUID to advertise

Documentation/fpga/xrt.rst line 32

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+   its compatibility requirement for child partition. When loading a child

partition the xmgmt

Documentation/fpga/xrt.rst line 33

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+   management driver matches parent UUID of the child partition against

child UUID exported by the

Documentation/fpga/xrt.rst line 34

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+   parent. Parent and child partition UUIDs are stored in the *xclbin* (for

user) or *xsabin* (for

Documentation/fpga/xrt.rst line 35

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+   base and shell). Except for root UUID, VSEC, hardware itself does not

know about UUIDs. UUIDs are

Quoted 19 lines

Documentation/fpga/xrt.rst new file +19 -0

+   stored in xsabin and xclbin.
+
+
+The physical partitions and their loading is illustrated below::
+
+            SHELL                               USER
+        +-----------+                  +-------------------+
+        |           |                  |                   |
+        | VSEC UUID | CHILD     PARENT |    LOGIC UUID     |
+        |           o------->|<--------o                   |
+        |           | UUID       UUID  |                   |
+        +-----+-----+                  +--------+----------+
+              |                                 |
+              .                                 .
+              |                                 |
+          +---+---+                      +------+--------+
+          |  POR  |                      | USER COMPILED |
+          | FLASH |                      |    XCLBIN     |
+          +-------+                      +---------------+

Documentation/fpga/xrt.rst

Quoted 7 lines

Documentation/fpga/xrt.rst new file +7 -0

+
+
+Loading Sequence
+----------------
+
+Shell partition is loaded from flash at system boot time. It
+establishes the PCIe link and exposes

Nit: The Shell

Documentation/fpga/xrt.rst line 67

Quoted 6 lines

Documentation/fpga/xrt.rst new file +6 -0

+two physical functions to the BIOS. After OS boot, xmgmt driver
+attaches to PCIe physical function
+0 exposed by the Shell and then looks for VSEC in PCIe extended
+configuration space. Using VSEC it determines the logic UUID of Shell
+and uses the UUID to load matching *xsabin* file from Linux firmware
+directory. The xsabin file contains metadata to discover peripherals that

are part of Shell and firmware(s) for any embedded soft processors in Shell.

Neat.

Documentation/fpga/xrt.rst line 70

Quoted 3 lines

Documentation/fpga/xrt.rst new file +3 -0

+
+Shell exports child interface UUID which is used for compatibility
+check when loading user compiled

Nit: The Shell

Documentation/fpga/xrt.rst line 74

Quoted 4 lines

Documentation/fpga/xrt.rst new file +4 -0

+xclbin over the User partition as part of DFX. When a user requests
+loading of a specific xclbin the xmgmt management driver reads the
+parent interface UUID specified in the xclbin and matches it with
+child interface UUID exported by Shell to determine if xclbin is compatible

with the Shell. If match fails loading of xclbin is denied.

Documentation/fpga/xrt.rst

Quoted 8 lines

Documentation/fpga/xrt.rst new file +8 -0

+
+xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl command.
+When loading xclbin xmgmt driver performs the following operations:
+
+1. Sanity check the xclbin contents
+2. Isolate the User partition
+3. Download the bitstream using the FPGA config engine (ICAP) 4.
+De-isolate the User partition

Is this modelled as bridges and regions?

Quoted 5 lines

Documentation/fpga/xrt.rst new file +4 -0

+5. Program the clocks (ClockWiz) driving the User partition 6. Wait
+for memory controller (MIG) calibration
+
+`Platform Loading Overview

Documentation/fpga/xrt.rst line 96

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+<https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`
+_ provides more detailed information on platform loading.
+
+xsabin
+------
+
+Each Alveo platform comes packaged with its own xsabin. The xsabin is
+trusted component of the platform. For format details refer to
+:ref:`xsabin/xclbin Container Format`. xsabin contains basic
+information like UUIDs, platform name and metadata in the form of

device tree. See :ref:Device Tree Usage for details and example.

Documentation/fpga/xrt.rst

Quoted 9 lines

Documentation/fpga/xrt.rst new file +9 -0

+
+xclbin
+------
+
+xclbin is compiled by end user using
+`Vitis
+<https://www.xilinx.com/products/design-tools/vitis/vitis-platform.ht
+ml>`_ tool set from Xilinx. The xclbin contains sections describing
+user compiled acceleration engines/kernels, memory subsystems,

clocking information etc. It also contains bitstream for the user partition, UUIDs, platform name, etc. xclbin uses the same container format as xsabin which is described below.

Documentation/fpga/xrt.rst

Quoted 6 lines

Documentation/fpga/xrt.rst new file +6 -0

+
+
+xsabin/xclbin Container Format
+------------------------------
+
+xclbin/xsabin is ELF-like binary container format. It is structured as series

of sections.

Documentation/fpga/xrt.rst line 112

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+There is a file header followed by several section headers which is

followed by sections.

Documentation/fpga/xrt.rst line 113

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+A section header points to an actual section. There is an optional signature

at the end.

Quoted 42 lines

Documentation/fpga/xrt.rst new file +42 -0

+The format is defined by header file ``xclbin.h``. The following
+figure illustrates a typical xclbin::
+
+
+          +---------------------+
+          |                     |
+          |       HEADER        |
+          +---------------------+
+          |   SECTION  HEADER   |
+          |                     |
+          +---------------------+
+          |         ...         |
+          |                     |
+          +---------------------+
+          |   SECTION  HEADER   |
+          |                     |
+          +---------------------+
+          |       SECTION       |
+          |                     |
+          +---------------------+
+          |         ...         |
+          |                     |
+          +---------------------+
+          |       SECTION       |
+          |                     |
+          +---------------------+
+          |      SIGNATURE      |
+          |      (OPTIONAL)     |
+          +---------------------+
+
+
+xclbin/xsabin files can be packaged, un-packaged and inspected using
+XRT utility called **xclbinutil**. xclbinutil is part of XRT open
+source software stack. The source code for xclbinutil can be found at
+https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbi
+nutil
+
+For example to enumerate the contents of a xclbin/xsabin use the
+*--info* switch as shown
+below::
+
+  xclbinutil --info --input

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+ /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
+  xclbinutil --info --input
+ /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsab
+ in
+
+
+Device Tree Usage
+-----------------
+
+As mentioned previously xsabin stores metadata which advertise HW

subsystems present in a partition.

Documentation/fpga/xrt.rst line 168

Quoted 3 lines

Documentation/fpga/xrt.rst new file +3 -0

+The metadata is stored in device tree format with well defined
+schema. Subsystem instantiations are captured as children of
+``addressable_endpoints`` node. Subsystem nodes have standard

attributes like ``reg``, ``interrupts`` etc. Additionally the nodes also have PCIe specific attributes:

Documentation/fpga/xrt.rst line 176

Quoted 8 lines

Documentation/fpga/xrt.rst new file +8 -0

+``pcie_physical_function`` and ``pcie_bar_mapping``. These identify
+which PCIe physical function and which BAR space in that physical
+function the subsystem resides. XRT management driver uses this
+information to bind *platform drivers* to the subsystem
+instantiations. The platform drivers are found in **xrt-lib.ko**
+kernel module defined later. Below is an example of device tree for
+Alveo U50
+platform::

I might be missing something, but couldn't you structure the addressable endpoints in a way that encode the physical function as a parent / child relation?

What are the regs relative to?

Quoted 14 lines

Documentation/fpga/xrt.rst new file +14 -0

+
+  /dts-v1/;
+
+  /{
+     logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
+
+     schema_version {
+             major = <0x01>;
+             minor = <0x00>;
+     };
+
+     interfaces {
+
+             @0 {

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+                     interface_uuid = "862c7020a250293e32036f19956669e5";
+             };
+     };
+
+     addressable_endpoints {
+
+             ep_blp_rom_00 {
+                     reg = <0x00 0x1f04000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_bram_ctrl-

1.0\0axi_bram_ctrl";

Documentation/fpga/xrt.rst

Quoted 6 lines

Documentation/fpga/xrt.rst new file +6 -0

+             };
+
+             ep_card_flash_program_00 {
+                     reg = <0x00 0x1f06000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_quad_spi-

1.0\0axi_quad_spi";

Quoted 49 lines

Documentation/fpga/xrt.rst new file +49 -0

+                     interrupts = <0x03 0x03>;
+             };
+
+             ep_cmc_firmware_mem_00 {
+                     reg = <0x00 0x1e20000 0x00 0x20000>;
+                     pcie_physical_function = <0x00>;
+                     compatible =
+ "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+                     firmware {
+                             firmware_product_name = "cmc";
+                             firmware_branch_name = "u50";
+                             firmware_version_major = <0x01>;
+                             firmware_version_minor = <0x00>;
+                     };
+             };
+
+             ep_cmc_intc_00 {
+                     reg = <0x00 0x1e03000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
+                     interrupts = <0x04 0x04>;
+             };
+
+             ep_cmc_mutex_00 {
+                     reg = <0x00 0x1e02000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+
+             ep_cmc_regmap_00 {
+                     reg = <0x00 0x1e08000 0x00 0x2000>;
+                     pcie_physical_function = <0x00>;
+                     compatible =
+ "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+                     firmware {
+                             firmware_product_name = "sc-fw";
+                             firmware_branch_name = "u50";
+                             firmware_version_major = <0x05>;
+                     };
+             };
+
+             ep_cmc_reset_00 {
+                     reg = <0x00 0x1e01000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+             ep_ddr_mem_calib_00 {
+                     reg = <0x00 0x63000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+
+             ep_debug_bscan_mgmt_00 {
+                     reg = <0x00 0x1e90000 0x00 0x10000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-debug_bridge-

1.0\0debug_bridge";

Documentation/fpga/xrt.rst

Quoted 12 lines

Documentation/fpga/xrt.rst new file +12 -0

+             };
+
+             ep_ert_base_address_00 {
+                     reg = <0x00 0x21000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+
+             ep_ert_command_queue_mgmt_00 {
+                     reg = <0x00 0x40000 0x00 0x10000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-ert_command_queue-

1.0\0ert_command_queue";

Documentation/fpga/xrt.rst

Quoted 6 lines

Documentation/fpga/xrt.rst new file +6 -0

+             };
+
+             ep_ert_command_queue_user_00 {
+                     reg = <0x00 0x40000 0x00 0x10000>;
+                     pcie_physical_function = <0x01>;
+                     compatible = "xilinx.com,reg_abs-ert_command_queue-

1.0\0ert_command_queue";

Quoted 81 lines

Documentation/fpga/xrt.rst new file +81 -0

+             };
+
+             ep_ert_firmware_mem_00 {
+                     reg = <0x00 0x30000 0x00 0x8000>;
+                     pcie_physical_function = <0x00>;
+                     compatible =
+ "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+                     firmware {
+                             firmware_product_name = "ert";
+                             firmware_branch_name = "v20";
+                             firmware_version_major = <0x01>;
+                     };
+             };
+
+             ep_ert_intc_00 {
+                     reg = <0x00 0x23000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
+                     interrupts = <0x05 0x05>;
+             };
+
+             ep_ert_reset_00 {
+                     reg = <0x00 0x22000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+
+             ep_ert_sched_00 {
+                     reg = <0x00 0x50000 0x00 0x1000>;
+                     pcie_physical_function = <0x01>;
+                     compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
+                     interrupts = <0x09 0x0c>;
+             };
+
+             ep_fpga_configuration_00 {
+                     reg = <0x00 0x1e88000 0x00 0x8000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
+                     interrupts = <0x02 0x02>;
+             };
+
+             ep_icap_reset_00 {
+                     reg = <0x00 0x1f07000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+
+             ep_mailbox_mgmt_00 {
+                     reg = <0x00 0x1f10000 0x00 0x10000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-mailbox-1.0\0mailbox";
+                     interrupts = <0x00 0x00>;
+             };
+
+             ep_mailbox_user_00 {
+                     reg = <0x00 0x1f00000 0x00 0x10000>;
+                     pcie_physical_function = <0x01>;
+                     compatible = "xilinx.com,reg_abs-mailbox-1.0\0mailbox";
+                     interrupts = <0x08 0x08>;
+             };
+
+             ep_msix_00 {
+                     reg = <0x00 0x00 0x00 0x20000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
+                     pcie_bar_mapping = <0x02>;
+             };
+
+             ep_pcie_link_mon_00 {
+                     reg = <0x00 0x1f05000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+
+             ep_pr_isolate_plp_00 {
+                     reg = <0x00 0x1f01000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+             ep_pr_isolate_ulp_00 {
+                     reg = <0x00 0x1000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+
+             ep_uuid_rom_00 {
+                     reg = <0x00 0x64000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_bram_ctrl-

1.0\0axi_bram_ctrl";

Quoted 13 lines

Documentation/fpga/xrt.rst new file +13 -0

+             };
+
+             ep_xdma_00 {
+                     reg = <0x00 0x00 0x00 0x10000>;
+                     pcie_physical_function = <0x01>;
+                     compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
+                     pcie_bar_mapping = <0x02>;
+             };
+     };
+
+  }
+
+

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+
+Deployment Models
+=================
+
+Baremetal
+---------
+
+In bare-metal deployments both MPF and UPF are visible and
+accessible. xmgmt driver binds to MPF. xmgmt driver operations are
+privileged and available to system administrator. The full stack is

illustrated below::

Quoted 24 lines

Documentation/fpga/xrt.rst new file +24 -0

+
+
+                            HOST
+
+                 [XMGMT]            [XUSER]
+                    |                  |
+                    |                  |
+                 +-----+            +-----+
+                 | MPF |            | UPF |
+                 |     |            |     |
+                 | PF0 |            | PF1 |
+                 +--+--+            +--+--+
+          ......... ^................. ^..........
+                    |                  |
+                    |   PCIe DEVICE    |
+                    |                  |
+                 +--+------------------+--+
+                 |         SHELL          |
+                 |                        |
+                 +------------------------+
+                 |         USER           |
+                 |                        |
+                 |                        |
+                 |                        |

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+                 |                        |
+                 +------------------------+
+
+
+
+Virtualized
+-----------
+
+In virtualized deployments privileged MPF is assigned to host but
+unprivileged UPF is assigned to guest VM via PCIe pass-through. xmgmt

driver in host binds to MPF.

Documentation/fpga/xrt.rst line 432

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+xmgmt driver operations are privileged and only accessible by hosting

service provider.

Quoted 40 lines

Documentation/fpga/xrt.rst new file +40 -0

+The full stack is illustrated below::
+
+
+                                 .............
+                  HOST           .    VM     .
+                                 .           .
+                 [XMGMT]         .  [XUSER]  .
+                    |            .     |     .
+                    |            .     |     .
+                 +-----+         .  +-----+  .
+                 | MPF |         .  | UPF |  .
+                 |     |         .  |     |  .
+                 | PF0 |         .  | PF1 |  .
+                 +--+--+         .  +--+--+  .
+          ......... ^................. ^..........
+                    |                  |
+                    |   PCIe DEVICE    |
+                    |                  |
+                 +--+------------------+--+
+                 |         SHELL          |
+                 |                        |
+                 +------------------------+
+                 |         USER           |
+                 |                        |
+                 |                        |
+                 |                        |
+                 |                        |
+                 +------------------------+
+
+
+
+Driver Modules
+==============
+
+xrt-lib.ko
+----------
+
+Repository of all subsystem drivers and pure software modules that
+can potentially be shared between xmgmt and xuser. All these drivers
+are structured as Linux *platform driver* and are instantiated by

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+xmgmt (or xuser in future) based on meta data associated with
+hardware. The metadata is in the form of device tree as explained before.
+
+xmgmt.ko
+--------
+
+The xmgmt driver is a PCIe device driver driving MPF found on
+Xilinx's Alveo PCIE device. It consists of one *root* driver, one or
+more *partition* drivers and one or more *leaf* drivers. The root and
+MPF specific leaf drivers are in xmgmt.ko. The partition driver and other

leaf drivers are in xrt-lib.ko.

Documentation/fpga/xrt.rst line 492

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+
+The instantiation of specific partition driver or leaf driver is
+completely data driven based on meta data (mostly in device tree
+format) found through VSEC capability and inside firmware files, such
+as xsabin or xclbin file. The root driver manages life cycle of
+multiple partition drivers, which, in turn, manages multiple leaf
+drivers. This allows a single set of driver code to support all kinds
+of subsystems exposed by different shells. The difference among all
+these subsystems will be handled in leaf drivers with root and
+partition drivers being part of the infrastructure and provide common

services for all leaves found on all platforms.

Documentation/fpga/xrt.rst

Quoted 7 lines

Documentation/fpga/xrt.rst new file +7 -0

+
+
+xmgmt-root
+^^^^^^^^^^
+
+The xmgmt-root driver is a PCIe device driver attaches to MPF. It's
+part of the

Nit: s/attaches/attached ?

Quoted 7 lines

Documentation/fpga/xrt.rst new file +7 -0

+infrastructure of the MPF driver and resides in xmgmt.ko. This driver
+
+* manages one or more partition drivers
+* provides access to functionalities that requires pci_dev, such as
+PCIE config
+  space access, to other leaf drivers through parent calls
+* together with partition driver, facilities event callbacks for

Documentation/fpga/xrt.rst line 516

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+other leaf drivers
+* together with partition driver, facilities inter-leaf driver calls
+for other leaf
+  drivers
+
+When root driver starts, it will explicitly create an initial
+partition instance, which contains leaf drivers that will trigger the
+creation of other partition instances. The root driver will wait for
+all partitions and leaves to be created before it returns from it's
+probe routine and claim success of the initialization of the entire xmgmt

driver.

Quoted 25 lines

Documentation/fpga/xrt.rst new file +25 -0

+
+partition
+^^^^^^^^^
+
+The partition driver is a platform device driver whose life cycle is
+managed by root and does not have real IO mem or IRQ resources. It's
+part of the infrastructure of the MPF driver and resides in
+xrt-lib.ko. This driver
+
+* manages one or more leaf drivers so that multiple leaves can be
+managed as a group
+* provides access to root from leaves, so that parent calls, event
+notifications
+  and inter-leaf calls can happen
+
+In xmgmt, an initial partition driver instance will be created by
+root, which contains leaves that will trigger partition instances to
+be created to manage groups of leaves found on different partitions
+on hardware, such as VSEC, Shell, and User.
+
+leaves
+^^^^^^
+
+The leaf driver is a platform device driver whose life cycle is
+managed by a partition driver and may or may not have real IO mem or

Documentation/fpga/xrt.rst line 551

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+IRQ resources. They are the real meat of xmgmt and contains platform
+specific code to Shell and User found on a MPF.
+
+A leaf driver may not have real hardware resources when it merely
+acts as a driver that manages certain in-memory states for xmgmt.
+These in-memory states could be shared by multiple other leaves.
+
+Leaf drivers assigned to specific hardware resources drive specific
+subsystem in the device. To manipulate the subsystem or carry out a
+task, a leaf driver may ask help from root via parent calls and/or from

other leaves via inter-leaf calls.

Documentation/fpga/xrt.rst line 555

Quoted 4 lines

Documentation/fpga/xrt.rst new file +4 -0

+
+A leaf can also broadcast events through infrastructure code for
+other leaves to process. It can also receive event notification from
+infrastructure about certain events, such as post-creation or pre-exit of a

particular leaf.

Documentation/fpga/xrt.rst

Quoted 9 lines

Documentation/fpga/xrt.rst new file +9 -0

+
+
+Driver Interfaces
+=================
+
+xmgmt Driver Ioctls
+-------------------
+
+Ioctls exposed by xmgmt driver to user space are enumerated in the

following table:

Documentation/fpga/xrt.rst line 566

Quoted 2 lines

Documentation/fpga/xrt.rst new file +2 -0

+
+== ===================== =============================

===========================

Documentation/fpga/xrt.rst line 568

Quoted 2 lines

Documentation/fpga/xrt.rst new file +2 -0

+#  Functionality         ioctl request code            data format
+== ===================== =============================

===========================

Documentation/fpga/xrt.rst line 569

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+1  FPGA image download   XMGMT_IOCICAPDOWNLOAD_AXLF

xmgmt_ioc_bitstream_axlf

Documentation/fpga/xrt.rst line 570

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+2  CL frequency scaling  XMGMT_IOCFREQSCALE

xmgmt_ioc_freqscaling

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+== ===================== =============================
+===========================
+
+xmgmt Driver Sysfs
+------------------
+
+xmgmt driver exposes a rich set of sysfs interfaces. Subsystem
+platform drivers export sysfs node for every platform instance.
+
+Every partition also exports its UUIDs. See below for examples::

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+
+  /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
+  /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
+
+
+hwmon
+-----
+
+xmgmt driver exposes standard hwmon interface to report voltage,
+current, temperature, power, etc. These can easily be viewed using

*sensors* command line utility.

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+
+
+mailbox
+-------
+
+xmgmt communicates with user physical function driver via HW mailbox.
+Mailbox opcodes are defined in ``mailbox_proto.h``. `Mailbox
+Inter-domain Communication Protocol
+<https://xilinx.github.io/XRT/master/html/mailbox.proto.html>`_
+defines the full specification. xmgmt implements subset of the

specification. It provides the following services to the UPF driver:

Quoted 2 lines

+
+1. Responding to *are you there* request including determining if the

two drivers are

Quoted 2 lines

+ running in the same OS domain
+2. Provide sensor readings, loaded xclbin UUID, clock frequency, shell

information, etc.

Quoted 2 lines

+3. Perform PCIe hot reset
+4. Download user compiled xclbin

Is this gonna use the mailbox framework?

Quoted 11 lines

Documentation/fpga/xrt.rst new file +8 -1

+
+
+Platform Security Considerations
+================================
+
+`Security of Alveo Platform
+<https://xilinx.github.io/XRT/master/html/security.html>`_
+discusses the deployment options and security implications in great detail.
--

2.17.1

That's a lot of text, I'll have to read it again most likely,

- Moritz

Hi Moritz,

Thanks for your feedback. Please see my reply inline.

Thanks,
-Max

Quoted 56 lines

-----Original Message-----
From: Moritz Fischer <[email protected]>
Sent: Monday, November 30, 2020 20:55
To: Sonal Santan <[email protected]>
Cc: [email protected]; [email protected]; Max Zhen <[email protected]>; Lizhi Hou <[email protected]>; Michal Simek <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers

Quoted 9 lines · Sonal Santan

From: Sonal Santan <[email protected]>

Describe Alveo XRT driver architecture and provide basic overview of Xilinx Alveo platform.

Signed-off-by Sonal Santan <[email protected]>

---
 Documentation/fpga/index.rst |   1 +
 Documentation/fpga/xrt.rst   | 588

+++++++++++++++++++++++++++++++++++

Quoted 5 lines

 2 files changed, 589 insertions(+)
 create mode 100644 Documentation/fpga/xrt.rst

(quoted diff) +0 -0

b/Documentation/fpga/index.rst index f80f95667ca2..30134357b70d

100644

Quoted 15 lines

Documentation/fpga/index.rst +1 -0

@@ -8,6 +8,7 @@ fpga
     :maxdepth: 1
 
     dfl
+    xrt
 
 .. only::  subproject and html
 

Documentation/fpga/xrt.rst new file +0 -0

new file mode 100644 index 000000000000..9f37d46459b0

Documentation/fpga/xrt.rst line 7

Quoted 7 lines

Documentation/fpga/xrt.rst new file +7 -0

@@ -0,1 +1,588 @@
+==================================
+XRTV2 Linux Kernel Driver Overview
+==================================
+
+XRTV2 drivers are second generation `XRT
+<https://github.com/Xilinx/XRT>`_ drivers which support `Alveo
+<https://www.xilinx.com/products/boards-and-kits/alveo.html>`_ PCIe

platforms from Xilinx.

Documentation/fpga/xrt.rst line 10

Quoted 3 lines

Documentation/fpga/xrt.rst new file +3 -0

+
+XRTV2 drivers support *subsystem* style data driven platforms where
+driver's configuration and behavior is determined by meta data provided

by platform (in *device tree* format).

Quoted 5 lines

Documentation/fpga/xrt.rst line 13

Quoted 3 lines

Documentation/fpga/xrt.rst new file +3 -0

+Primary management physical function (MPF) driver is called
+**xmgmt**. Primary user physical function (UPF) driver is called
+**xuser** and HW subsystem drivers are packaged into a library module

called **xrt-lib**, which is shared by **xmgmt** and **xuser** (WIP). WIP?

Working in progress. I'll expand it in the doc.

Quoted 7 lines

Documentation/fpga/xrt.rst

Quoted 6 lines

Documentation/fpga/xrt.rst new file +6 -0

+
+Alveo Platform Overview
+=======================
+
+Alveo platforms are architected as two physical FPGA partitions:
+*Shell* and *User*. Shell

Nit: The Shell provides ...

Sure. Will fix.

Quoted 45 lines

Documentation/fpga/xrt.rst line 22

Quoted 3 lines

Documentation/fpga/xrt.rst new file +3 -0

+provides basic infrastructure for the Alveo platform like PCIe
+connectivity, board management, Dynamic Function Exchange (DFX),
+sensors, clocking, reset, and security. User partition contains user

compiled binary which is loaded by a process called DFX also known as partial reconfiguration.

Documentation/fpga/xrt.rst line 24

Quoted 2 lines

Documentation/fpga/xrt.rst new file +2 -0

+
+Physical partitions require strict HW compatibility with each other for DFX

to work properly.

Documentation/fpga/xrt.rst line 27

Quoted 3 lines

Documentation/fpga/xrt.rst new file +3 -0

+Every physical partition has two interface UUIDs: *parent* UUID and
+*child* UUID. For simple single stage platforms Shell → User forms
+parent child relationship. For complex two stage platforms Base → Shell

→ User forms the parent child relationship chain.

Documentation/fpga/xrt.rst

Quoted 3 lines

Documentation/fpga/xrt.rst new file +3 -0

+
+.. note::
+   Partition compatibility matching is key design component of Alveo

platforms and XRT. Partitions

Documentation/fpga/xrt.rst line 31

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+   have child and parent relationship. A loaded partition exposes child

partition UUID to advertise

Documentation/fpga/xrt.rst line 32

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+   its compatibility requirement for child partition. When loading a child

partition the xmgmt

Documentation/fpga/xrt.rst line 33

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+   management driver matches parent UUID of the child partition against

child UUID exported by the

Documentation/fpga/xrt.rst line 34

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+   parent. Parent and child partition UUIDs are stored in the *xclbin* (for

user) or *xsabin* (for

Documentation/fpga/xrt.rst line 35

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+   base and shell). Except for root UUID, VSEC, hardware itself does not

know about UUIDs. UUIDs are

Quoted 19 lines

Documentation/fpga/xrt.rst new file +19 -0

+   stored in xsabin and xclbin.
+
+
+The physical partitions and their loading is illustrated below::
+
+            SHELL                               USER
+        +-----------+                  +-------------------+
+        |           |                  |                   |
+        | VSEC UUID | CHILD     PARENT |    LOGIC UUID     |
+        |           o------->|<--------o                   |
+        |           | UUID       UUID  |                   |
+        +-----+-----+                  +--------+----------+
+              |                                 |
+              .                                 .
+              |                                 |
+          +---+---+                      +------+--------+
+          |  POR  |                      | USER COMPILED |
+          | FLASH |                      |    XCLBIN     |
+          +-------+                      +---------------+

Quoted 8 lines

Documentation/fpga/xrt.rst

Quoted 7 lines

Documentation/fpga/xrt.rst new file +7 -0

+
+
+Loading Sequence
+----------------
+
+Shell partition is loaded from flash at system boot time. It
+establishes the PCIe link and exposes

Nit: The Shell

Will fix.

Quoted 9 lines

Documentation/fpga/xrt.rst line 67

Quoted 6 lines

Documentation/fpga/xrt.rst new file +6 -0

+two physical functions to the BIOS. After OS boot, xmgmt driver
+attaches to PCIe physical function
+0 exposed by the Shell and then looks for VSEC in PCIe extended
+configuration space. Using VSEC it determines the logic UUID of Shell
+and uses the UUID to load matching *xsabin* file from Linux firmware
+directory. The xsabin file contains metadata to discover peripherals that

are part of Shell and firmware(s) for any embedded soft processors in Shell.

Neat.

Thanks :-).

Quoted 4 lines

Documentation/fpga/xrt.rst line 70

Quoted 3 lines

Documentation/fpga/xrt.rst new file +3 -0

+
+Shell exports child interface UUID which is used for compatibility
+check when loading user compiled

Nit: The Shell

Sure.

Quoted 5 lines

Documentation/fpga/xrt.rst line 74

Quoted 4 lines

Documentation/fpga/xrt.rst new file +4 -0

+xclbin over the User partition as part of DFX. When a user requests
+loading of a specific xclbin the xmgmt management driver reads the
+parent interface UUID specified in the xclbin and matches it with
+child interface UUID exported by Shell to determine if xclbin is compatible

with the Shell. If match fails loading of xclbin is denied.

Quoted 9 lines

Documentation/fpga/xrt.rst

Quoted 8 lines

Documentation/fpga/xrt.rst new file +8 -0

+
+xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl command.
+When loading xclbin xmgmt driver performs the following operations:
+
+1. Sanity check the xclbin contents
+2. Isolate the User partition
+3. Download the bitstream using the FPGA config engine (ICAP) 4.
+De-isolate the User partition

Is this modelled as bridges and regions?

Alveo drivers as written today do not use fpga bridge and region framework. It seems that if we add support for that framework, it’s possible to receive PR program request from kernel outside of xmgmt driver? Currently, we can’t support this and PR program can only be initiated using XRT’s runtime API in user space.

Or maybe we have missed some points about the use case for this framework?

Quoted 97 lines

Quoted 5 lines

Documentation/fpga/xrt.rst new file +4 -0

+5. Program the clocks (ClockWiz) driving the User partition 6. Wait
+for memory controller (MIG) calibration
+
+`Platform Loading Overview

Documentation/fpga/xrt.rst line 96

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+<https://xilinx.github.io/XRT/master/html/platforms_partitions.html>`
+_ provides more detailed information on platform loading.
+
+xsabin
+------
+
+Each Alveo platform comes packaged with its own xsabin. The xsabin is
+trusted component of the platform. For format details refer to
+:ref:`xsabin/xclbin Container Format`. xsabin contains basic
+information like UUIDs, platform name and metadata in the form of

device tree. See :ref:Device Tree Usage for details and example.

Documentation/fpga/xrt.rst

Quoted 9 lines

Documentation/fpga/xrt.rst new file +9 -0

+
+xclbin
+------
+
+xclbin is compiled by end user using
+`Vitis
+<https://www.xilinx.com/products/design-tools/vitis/vitis-platform.ht
+ml>`_ tool set from Xilinx. The xclbin contains sections describing
+user compiled acceleration engines/kernels, memory subsystems,

clocking information etc. It also contains bitstream for the user partition, UUIDs, platform name, etc. xclbin uses the same container format as xsabin which is described below.

Documentation/fpga/xrt.rst

Quoted 6 lines

Documentation/fpga/xrt.rst new file +6 -0

+
+
+xsabin/xclbin Container Format
+------------------------------
+
+xclbin/xsabin is ELF-like binary container format. It is structured as series

of sections.

Documentation/fpga/xrt.rst line 112

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+There is a file header followed by several section headers which is

followed by sections.

Documentation/fpga/xrt.rst line 113

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+A section header points to an actual section. There is an optional signature

at the end.

Quoted 42 lines

Documentation/fpga/xrt.rst new file +42 -0

+The format is defined by header file ``xclbin.h``. The following
+figure illustrates a typical xclbin::
+
+
+          +---------------------+
+          |                     |
+          |       HEADER        |
+          +---------------------+
+          |   SECTION  HEADER   |
+          |                     |
+          +---------------------+
+          |         ...         |
+          |                     |
+          +---------------------+
+          |   SECTION  HEADER   |
+          |                     |
+          +---------------------+
+          |       SECTION       |
+          |                     |
+          +---------------------+
+          |         ...         |
+          |                     |
+          +---------------------+
+          |       SECTION       |
+          |                     |
+          +---------------------+
+          |      SIGNATURE      |
+          |      (OPTIONAL)     |
+          +---------------------+
+
+
+xclbin/xsabin files can be packaged, un-packaged and inspected using
+XRT utility called **xclbinutil**. xclbinutil is part of XRT open
+source software stack. The source code for xclbinutil can be found at
+https://github.com/Xilinx/XRT/tree/master/src/runtime_src/tools/xclbi
+nutil
+
+For example to enumerate the contents of a xclbin/xsabin use the
+*--info* switch as shown
+below::
+
+  xclbinutil --info --input

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+ /opt/xilinx/firmware/u50/gen3x16-xdma/blp/test/bandwidth.xclbin
+  xclbinutil --info --input
+ /lib/firmware/xilinx/862c7020a250293e32036f19956669e5/partition.xsab
+ in
+
+
+Device Tree Usage
+-----------------
+
+As mentioned previously xsabin stores metadata which advertise HW

subsystems present in a partition.

Documentation/fpga/xrt.rst line 168

Quoted 3 lines

Documentation/fpga/xrt.rst new file +3 -0

+The metadata is stored in device tree format with well defined
+schema. Subsystem instantiations are captured as children of
+``addressable_endpoints`` node. Subsystem nodes have standard

attributes like ``reg``, ``interrupts`` etc. Additionally the nodes also have PCIe specific attributes:

Quoted 12 lines

Documentation/fpga/xrt.rst line 176

Quoted 8 lines

Documentation/fpga/xrt.rst new file +8 -0

+``pcie_physical_function`` and ``pcie_bar_mapping``. These identify
+which PCIe physical function and which BAR space in that physical
+function the subsystem resides. XRT management driver uses this
+information to bind *platform drivers* to the subsystem
+instantiations. The platform drivers are found in **xrt-lib.ko**
+kernel module defined later. Below is an example of device tree for
+Alveo U50
+platform::

I might be missing something, but couldn't you structure the addressable endpoints in a way that encode the physical function as a parent / child relation?

Alveo driver does not generate the metadata. The metadata is formatted and generated by HW tools when the Alveo HW platform is built.

Quoted 3 lines

What are the regs relative to?

Regs indicates offset of the register on the PCIE BAR of the Alveo device.

Quoted 327 lines

Quoted 14 lines

Documentation/fpga/xrt.rst new file +14 -0

+
+  /dts-v1/;
+
+  /{
+     logic_uuid = "f465b0a3ae8c64f619bc150384ace69b";
+
+     schema_version {
+             major = <0x01>;
+             minor = <0x00>;
+     };
+
+     interfaces {
+
+             @0 {

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+                     interface_uuid = "862c7020a250293e32036f19956669e5";
+             };
+     };
+
+     addressable_endpoints {
+
+             ep_blp_rom_00 {
+                     reg = <0x00 0x1f04000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_bram_ctrl-

1.0\0axi_bram_ctrl";

Documentation/fpga/xrt.rst

Quoted 6 lines

Documentation/fpga/xrt.rst new file +6 -0

+             };
+
+             ep_card_flash_program_00 {
+                     reg = <0x00 0x1f06000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_quad_spi-

1.0\0axi_quad_spi";

Quoted 49 lines

Documentation/fpga/xrt.rst new file +49 -0

+                     interrupts = <0x03 0x03>;
+             };
+
+             ep_cmc_firmware_mem_00 {
+                     reg = <0x00 0x1e20000 0x00 0x20000>;
+                     pcie_physical_function = <0x00>;
+                     compatible =
+ "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+                     firmware {
+                             firmware_product_name = "cmc";
+                             firmware_branch_name = "u50";
+                             firmware_version_major = <0x01>;
+                             firmware_version_minor = <0x00>;
+                     };
+             };
+
+             ep_cmc_intc_00 {
+                     reg = <0x00 0x1e03000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
+                     interrupts = <0x04 0x04>;
+             };
+
+             ep_cmc_mutex_00 {
+                     reg = <0x00 0x1e02000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+
+             ep_cmc_regmap_00 {
+                     reg = <0x00 0x1e08000 0x00 0x2000>;
+                     pcie_physical_function = <0x00>;
+                     compatible =
+ "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+                     firmware {
+                             firmware_product_name = "sc-fw";
+                             firmware_branch_name = "u50";
+                             firmware_version_major = <0x05>;
+                     };
+             };
+
+             ep_cmc_reset_00 {
+                     reg = <0x00 0x1e01000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+             ep_ddr_mem_calib_00 {
+                     reg = <0x00 0x63000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+
+             ep_debug_bscan_mgmt_00 {
+                     reg = <0x00 0x1e90000 0x00 0x10000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-debug_bridge-

1.0\0debug_bridge";

Documentation/fpga/xrt.rst

Quoted 12 lines

Documentation/fpga/xrt.rst new file +12 -0

+             };
+
+             ep_ert_base_address_00 {
+                     reg = <0x00 0x21000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+
+             ep_ert_command_queue_mgmt_00 {
+                     reg = <0x00 0x40000 0x00 0x10000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-ert_command_queue-

1.0\0ert_command_queue";

Documentation/fpga/xrt.rst

Quoted 6 lines

Documentation/fpga/xrt.rst new file +6 -0

+             };
+
+             ep_ert_command_queue_user_00 {
+                     reg = <0x00 0x40000 0x00 0x10000>;
+                     pcie_physical_function = <0x01>;
+                     compatible = "xilinx.com,reg_abs-ert_command_queue-

1.0\0ert_command_queue";

Quoted 81 lines

Documentation/fpga/xrt.rst new file +81 -0

+             };
+
+             ep_ert_firmware_mem_00 {
+                     reg = <0x00 0x30000 0x00 0x8000>;
+                     pcie_physical_function = <0x00>;
+                     compatible =
+ "xilinx.com,reg_abs-axi_bram_ctrl-1.0\0axi_bram_ctrl";
+
+                     firmware {
+                             firmware_product_name = "ert";
+                             firmware_branch_name = "v20";
+                             firmware_version_major = <0x01>;
+                     };
+             };
+
+             ep_ert_intc_00 {
+                     reg = <0x00 0x23000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_intc-1.0\0axi_intc";
+                     interrupts = <0x05 0x05>;
+             };
+
+             ep_ert_reset_00 {
+                     reg = <0x00 0x22000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+
+             ep_ert_sched_00 {
+                     reg = <0x00 0x50000 0x00 0x1000>;
+                     pcie_physical_function = <0x01>;
+                     compatible = "xilinx.com,reg_abs-ert_sched-1.0\0ert_sched";
+                     interrupts = <0x09 0x0c>;
+             };
+
+             ep_fpga_configuration_00 {
+                     reg = <0x00 0x1e88000 0x00 0x8000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_hwicap-1.0\0axi_hwicap";
+                     interrupts = <0x02 0x02>;
+             };
+
+             ep_icap_reset_00 {
+                     reg = <0x00 0x1f07000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+
+             ep_mailbox_mgmt_00 {
+                     reg = <0x00 0x1f10000 0x00 0x10000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-mailbox-1.0\0mailbox";
+                     interrupts = <0x00 0x00>;
+             };
+
+             ep_mailbox_user_00 {
+                     reg = <0x00 0x1f00000 0x00 0x10000>;
+                     pcie_physical_function = <0x01>;
+                     compatible = "xilinx.com,reg_abs-mailbox-1.0\0mailbox";
+                     interrupts = <0x08 0x08>;
+             };
+
+             ep_msix_00 {
+                     reg = <0x00 0x00 0x00 0x20000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-msix-1.0\0msix";
+                     pcie_bar_mapping = <0x02>;
+             };
+
+             ep_pcie_link_mon_00 {
+                     reg = <0x00 0x1f05000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+
+             ep_pr_isolate_plp_00 {
+                     reg = <0x00 0x1f01000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+             ep_pr_isolate_ulp_00 {
+                     reg = <0x00 0x1000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_gpio-1.0\0axi_gpio";
+             };
+
+             ep_uuid_rom_00 {
+                     reg = <0x00 0x64000 0x00 0x1000>;
+                     pcie_physical_function = <0x00>;
+                     compatible = "xilinx.com,reg_abs-axi_bram_ctrl-

1.0\0axi_bram_ctrl";

Quoted 13 lines

Documentation/fpga/xrt.rst new file +13 -0

+             };
+
+             ep_xdma_00 {
+                     reg = <0x00 0x00 0x00 0x10000>;
+                     pcie_physical_function = <0x01>;
+                     compatible = "xilinx.com,reg_abs-xdma-1.0\0xdma";
+                     pcie_bar_mapping = <0x02>;
+             };
+     };
+
+  }
+
+

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+
+Deployment Models
+=================
+
+Baremetal
+---------
+
+In bare-metal deployments both MPF and UPF are visible and
+accessible. xmgmt driver binds to MPF. xmgmt driver operations are
+privileged and available to system administrator. The full stack is

illustrated below::

Quoted 24 lines

Documentation/fpga/xrt.rst new file +24 -0

+
+
+                            HOST
+
+                 [XMGMT]            [XUSER]
+                    |                  |
+                    |                  |
+                 +-----+            +-----+
+                 | MPF |            | UPF |
+                 |     |            |     |
+                 | PF0 |            | PF1 |
+                 +--+--+            +--+--+
+          ......... ^................. ^..........
+                    |                  |
+                    |   PCIe DEVICE    |
+                    |                  |
+                 +--+------------------+--+
+                 |         SHELL          |
+                 |                        |
+                 +------------------------+
+                 |         USER           |
+                 |                        |
+                 |                        |
+                 |                        |

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+                 |                        |
+                 +------------------------+
+
+
+
+Virtualized
+-----------
+
+In virtualized deployments privileged MPF is assigned to host but
+unprivileged UPF is assigned to guest VM via PCIe pass-through. xmgmt

driver in host binds to MPF.

Documentation/fpga/xrt.rst line 432

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+xmgmt driver operations are privileged and only accessible by hosting

service provider.

Quoted 40 lines

Documentation/fpga/xrt.rst new file +40 -0

+The full stack is illustrated below::
+
+
+                                 .............
+                  HOST           .    VM     .
+                                 .           .
+                 [XMGMT]         .  [XUSER]  .
+                    |            .     |     .
+                    |            .     |     .
+                 +-----+         .  +-----+  .
+                 | MPF |         .  | UPF |  .
+                 |     |         .  |     |  .
+                 | PF0 |         .  | PF1 |  .
+                 +--+--+         .  +--+--+  .
+          ......... ^................. ^..........
+                    |                  |
+                    |   PCIe DEVICE    |
+                    |                  |
+                 +--+------------------+--+
+                 |         SHELL          |
+                 |                        |
+                 +------------------------+
+                 |         USER           |
+                 |                        |
+                 |                        |
+                 |                        |
+                 |                        |
+                 +------------------------+
+
+
+
+Driver Modules
+==============
+
+xrt-lib.ko
+----------
+
+Repository of all subsystem drivers and pure software modules that
+can potentially be shared between xmgmt and xuser. All these drivers
+are structured as Linux *platform driver* and are instantiated by

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+xmgmt (or xuser in future) based on meta data associated with
+hardware. The metadata is in the form of device tree as explained before.
+
+xmgmt.ko
+--------
+
+The xmgmt driver is a PCIe device driver driving MPF found on
+Xilinx's Alveo PCIE device. It consists of one *root* driver, one or
+more *partition* drivers and one or more *leaf* drivers. The root and
+MPF specific leaf drivers are in xmgmt.ko. The partition driver and other

leaf drivers are in xrt-lib.ko.

Documentation/fpga/xrt.rst line 492

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+
+The instantiation of specific partition driver or leaf driver is
+completely data driven based on meta data (mostly in device tree
+format) found through VSEC capability and inside firmware files, such
+as xsabin or xclbin file. The root driver manages life cycle of
+multiple partition drivers, which, in turn, manages multiple leaf
+drivers. This allows a single set of driver code to support all kinds
+of subsystems exposed by different shells. The difference among all
+these subsystems will be handled in leaf drivers with root and
+partition drivers being part of the infrastructure and provide common

services for all leaves found on all platforms.

Quoted 8 lines

Documentation/fpga/xrt.rst

Quoted 7 lines

Documentation/fpga/xrt.rst new file +7 -0

+
+
+xmgmt-root
+^^^^^^^^^^
+
+The xmgmt-root driver is a PCIe device driver attaches to MPF. It's
+part of the

Nit: s/attaches/attached ?

Yes, sure.

Quoted 120 lines

Quoted 7 lines

Documentation/fpga/xrt.rst new file +7 -0

+infrastructure of the MPF driver and resides in xmgmt.ko. This driver
+
+* manages one or more partition drivers
+* provides access to functionalities that requires pci_dev, such as
+PCIE config
+  space access, to other leaf drivers through parent calls
+* together with partition driver, facilities event callbacks for

Documentation/fpga/xrt.rst line 516

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+other leaf drivers
+* together with partition driver, facilities inter-leaf driver calls
+for other leaf
+  drivers
+
+When root driver starts, it will explicitly create an initial
+partition instance, which contains leaf drivers that will trigger the
+creation of other partition instances. The root driver will wait for
+all partitions and leaves to be created before it returns from it's
+probe routine and claim success of the initialization of the entire xmgmt

driver.

Quoted 25 lines

Documentation/fpga/xrt.rst new file +25 -0

+
+partition
+^^^^^^^^^
+
+The partition driver is a platform device driver whose life cycle is
+managed by root and does not have real IO mem or IRQ resources. It's
+part of the infrastructure of the MPF driver and resides in
+xrt-lib.ko. This driver
+
+* manages one or more leaf drivers so that multiple leaves can be
+managed as a group
+* provides access to root from leaves, so that parent calls, event
+notifications
+  and inter-leaf calls can happen
+
+In xmgmt, an initial partition driver instance will be created by
+root, which contains leaves that will trigger partition instances to
+be created to manage groups of leaves found on different partitions
+on hardware, such as VSEC, Shell, and User.
+
+leaves
+^^^^^^
+
+The leaf driver is a platform device driver whose life cycle is
+managed by a partition driver and may or may not have real IO mem or

Documentation/fpga/xrt.rst line 551

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+IRQ resources. They are the real meat of xmgmt and contains platform
+specific code to Shell and User found on a MPF.
+
+A leaf driver may not have real hardware resources when it merely
+acts as a driver that manages certain in-memory states for xmgmt.
+These in-memory states could be shared by multiple other leaves.
+
+Leaf drivers assigned to specific hardware resources drive specific
+subsystem in the device. To manipulate the subsystem or carry out a
+task, a leaf driver may ask help from root via parent calls and/or from

other leaves via inter-leaf calls.

Documentation/fpga/xrt.rst line 555

Quoted 4 lines

Documentation/fpga/xrt.rst new file +4 -0

+
+A leaf can also broadcast events through infrastructure code for
+other leaves to process. It can also receive event notification from
+infrastructure about certain events, such as post-creation or pre-exit of a

particular leaf.

Documentation/fpga/xrt.rst

Quoted 9 lines

Documentation/fpga/xrt.rst new file +9 -0

+
+
+Driver Interfaces
+=================
+
+xmgmt Driver Ioctls
+-------------------
+
+Ioctls exposed by xmgmt driver to user space are enumerated in the

following table:

Documentation/fpga/xrt.rst line 566

Quoted 2 lines

Documentation/fpga/xrt.rst new file +2 -0

+
+== ===================== =============================

===========================

Documentation/fpga/xrt.rst line 568

Quoted 2 lines

Documentation/fpga/xrt.rst new file +2 -0

+#  Functionality         ioctl request code            data format
+== ===================== =============================

===========================

Documentation/fpga/xrt.rst line 569

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+1  FPGA image download   XMGMT_IOCICAPDOWNLOAD_AXLF

xmgmt_ioc_bitstream_axlf

Documentation/fpga/xrt.rst line 570

Quoted 1 line

Documentation/fpga/xrt.rst new file +1 -0

+2  CL frequency scaling  XMGMT_IOCFREQSCALE

xmgmt_ioc_freqscaling

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+== ===================== =============================
+===========================
+
+xmgmt Driver Sysfs
+------------------
+
+xmgmt driver exposes a rich set of sysfs interfaces. Subsystem
+platform drivers export sysfs node for every platform instance.
+
+Every partition also exports its UUIDs. See below for examples::

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+
+  /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/interface_uuids
+  /sys/bus/pci/devices/0000:06:00.0/xmgmt_main.0/logic_uuids
+
+
+hwmon
+-----
+
+xmgmt driver exposes standard hwmon interface to report voltage,
+current, temperature, power, etc. These can easily be viewed using

*sensors* command line utility.

Documentation/fpga/xrt.rst

Quoted 10 lines

Documentation/fpga/xrt.rst new file +10 -0

+
+
+mailbox
+-------
+
+xmgmt communicates with user physical function driver via HW mailbox.
+Mailbox opcodes are defined in ``mailbox_proto.h``. `Mailbox
+Inter-domain Communication Protocol
+<https://xilinx.github.io/XRT/master/html/mailbox.proto.html>`_
+defines the full specification. xmgmt implements subset of the

specification. It provides the following services to the UPF driver:

Quoted 2 lines

+
+1. Responding to *are you there* request including determining if the

two drivers are

Quoted 2 lines

+ running in the same OS domain
+2. Provide sensor readings, loaded xclbin UUID, clock frequency, shell

information, etc.

Quoted 2 lines

+3. Perform PCIe hot reset
+4. Download user compiled xclbin

Quoted 1 line

Is this gonna use the mailbox framework?

The xclbin can be downloaded via IOCTL interface of xmgmt driver. Or the download request can come from user pf driver via mailbox, yes.

Thanks,
Max

Quoted 15 lines

Quoted 11 lines

Documentation/fpga/xrt.rst new file +8 -1

+
+
+Platform Security Considerations
+================================
+
+`Security of Alveo Platform
+<https://xilinx.github.io/XRT/master/html/security.html>`_
+discusses the deployment options and security implications in great detail.
--

2.17.1

That's a lot of text, I'll have to read it again most likely,

- Moritz

-----Original Message-----
From: Moritz Fischer <[email protected]>
Sent: Wednesday, December 2, 2020 15:10
To: Max Zhen <[email protected]>
Cc: Moritz Fischer <[email protected]>; Sonal Santan <[email protected]>; [email protected]; [email protected]; Lizhi Hou <[email protected]>; Michal Simek <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers

Hi Max,

Quoted 33 lines · Max Zhen

Hi Moritz,

Thanks for your feedback. Please see my reply inline.

Thanks,
-Max

Quoted 26 lines

-----Original Message-----
From: Moritz Fischer <[email protected]>
Sent: Monday, November 30, 2020 20:55
To: Sonal Santan <[email protected]>
Cc: [email protected]; [email protected]; Max Zhen <[email protected]>; Lizhi Hou <[email protected]>; Michal Simek <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH Xilinx Alveo 1/8] Documentation: fpga: Add a document describing Alveo XRT drivers

Quoted 9 lines · Sonal Santan

From: Sonal Santan <[email protected]>

Describe Alveo XRT driver architecture and provide basic overview of Xilinx Alveo platform.

Signed-off-by Sonal Santan <[email protected]>

---
 Documentation/fpga/index.rst |   1 +
 Documentation/fpga/xrt.rst   | 588

+++++++++++++++++++++++++++++++++++

Quoted 3 lines

 2 files changed, 589 insertions(+)  create mode 100644 

Documentation/fpga/xrt.rst

Quoted 8 lines

Quoted 8 lines

Review

Quoted 5 lines

(quoted diff) +5 -0

+xclbin over the User partition as part of DFX. When a user 
+requests loading of a specific xclbin the xmgmt management 
+driver reads the parent interface UUID specified in the xclbin 
+and matches it with child interface UUID exported by Shell to 
+determine if xclbin is compatible

with the Shell. If match fails loading of xclbin is denied.

Review

Quoted 2 lines

(quoted diff) +2 -0

+
+xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl

command.

Quoted 8 lines

Quoted 6 lines

Review

Quoted 5 lines

(quoted diff) +5 -0

+When loading xclbin xmgmt driver performs the following operations:
+
+1. Sanity check the xclbin contents 2. Isolate the User 
+partition 3. Download the bitstream using the FPGA config engine (ICAP) 4.
+De-isolate the User partition

Is this modelled as bridges and regions?

Alveo drivers as written today do not use fpga bridge and region

framework. It seems that if we add support for that framework, it’s possible to receive PR program request from kernel outside of xmgmt driver? Currently, we can’t support this and PR program can only be initiated using XRT’s runtime API in user space.

I'm not 100% sure I understand the concern here, let me reply to what I think I understand:

You're worried that if you use FPGA region as interface to accept PR requests something else could attempt to reconfigure the region from within the kernel using the FPGA Region API?

Assuming I got this right, I don't think this is a big deal. When you create the regions you control who gets the references to it.

From what I've seen so far Regions seem to be roughly equivalent to Partitions, hence my surprise to see a new structure bypassing them.

Quoted 2 lines

Or maybe we have missed some points about the use case for this

framework?

Quoted 1 line

Quoted 23 lines

Quoted 22 lines

Review

Quoted 4 lines

+-----------------

(quoted diff) +3 -0

+
+As mentioned previously xsabin stores metadata which advertise 
+HW

subsystems present in a partition.

Review

Quoted 3 lines

(quoted diff) +3 -0

+The metadata is stored in device tree format with well defined 
+schema. Subsystem instantiations are captured as children of 
+``addressable_endpoints`` node. Subsystem nodes have standard

attributes like ``reg``, ``interrupts`` etc. Additionally the nodes also have PCIe specific attributes:

Review

Quoted 8 lines

+``pcie_physical_function`` and ``pcie_bar_mapping``. These

(quoted diff) +7 -0

+identify which PCIe physical function and which BAR space in 
+that physical function the subsystem resides. XRT management 
+driver uses this information to bind *platform drivers* to the 
+subsystem instantiations. The platform drivers are found in 
+**xrt-lib.ko** kernel module defined later. Below is an example 
+of device tree for Alveo U50
+platform::

I might be missing something, but couldn't you structure the addressable endpoints in a way that encode the physical function as a parent / child relation?

Quoted 2 lines

Alveo driver does not generate the metadata. The metadata is formatted

and generated by HW tools when the Alveo HW platform is built.

Sure, but you control the tools that generate the metadata :) Your userland can structure / process it however it wants / needs?

[...cut...]

Quoted 25 lines

Quoted 8 lines

Quoted 8 lines

Review

Quoted 5 lines

(quoted diff) +5 -0

+xclbin over the User partition as part of DFX. When a user 
+requests loading of a specific xclbin the xmgmt management 
+driver reads the parent interface UUID specified in the xclbin 
+and matches it with child interface UUID exported by Shell to 
+determine if xclbin is compatible

with the Shell. If match fails loading of xclbin is denied.

Review

Quoted 2 lines

(quoted diff) +2 -0

+
+xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl

command.

Quoted 8 lines

Quoted 6 lines

Review

Quoted 5 lines

(quoted diff) +5 -0

+When loading xclbin xmgmt driver performs the following operations:
+
+1. Sanity check the xclbin contents 2. Isolate the User 
+partition 3. Download the bitstream using the FPGA config engine (ICAP) 4.
+De-isolate the User partition

Is this modelled as bridges and regions?

Alveo drivers as written today do not use fpga bridge and region

framework. It seems that if we add support for that framework, it’s possible to receive PR program request from kernel outside of xmgmt driver? Currently, we can’t support this and PR program can only be initiated using XRT’s runtime API in user space.

I'm not 100% sure I understand the concern here, let me reply to what I think I understand:

Quoted 6 lines

You're worried that if you use FPGA region as interface to accept PR requests something else could attempt to reconfigure the region from within the kernel using the FPGA Region API?

Assuming I got this right, I don't think this is a big deal. When you create the regions you control who gets the references to it.

Thanks for explaining. Yes, I think you got my point :-).

Quoted 4 lines

From what I've seen so far Regions seem to be roughly equivalent to Partitions, hence my surprise to see a new structure bypassing them.

I see where the gap is.

Regions in Linux is very different than "partitions" we have defined in xmgmt. Regions seem to be a software data structure representing an area on the FPGA that can be reprogrammed. This area is protected by the concept of "bridge" which can be disabled before program and reenabled after it. And you go through region when you need to reprogram this area.

The "partition" is part of the main infrastructure of xmgmt driver, which represents a group of subdev drivers for each individual IP (HW subcomponents). Basically, xmgmt root driver is parent of several partitions who is, in turn, the parent of several subdev drivers. The parent manages the life cycle of its children here.

We do have a partition to represent the group of subdevs/IPs in the reprogrammable area. And we also have partitions representing other areas which cannot be reprogrammed. So, it is difficult to use "Region" to implement "partition".

From what you have explained, it seems that even if I use region / bridge in xmgmt, we can still keep it private to xmgmt instead of exposing the interface to outside world, which we can't support anyway? This means that region will be used as an internal data structure for xmgmt. Since we can't simply replace partition with region, we might as well just use partition throughout the driver, instead of introducing two data structures and use them both in different places.

However, if using region/bridge can bring in other benefits, please let us know and we could see if we can also add this to xmgmt.

Quoted 5 lines

Quoted 3 lines

Or maybe we have missed some points about the use case for this

framework?

Quoted 1 line

[...cut...]

Quoted 23 lines

Quoted 23 lines

Quoted 22 lines

Review

Quoted 4 lines

+-----------------

(quoted diff) +3 -0

+
+As mentioned previously xsabin stores metadata which advertise 
+HW

subsystems present in a partition.

Review

Quoted 3 lines

(quoted diff) +3 -0

+The metadata is stored in device tree format with well defined 
+schema. Subsystem instantiations are captured as children of 
+``addressable_endpoints`` node. Subsystem nodes have standard

attributes like ``reg``, ``interrupts`` etc. Additionally the nodes also have PCIe specific attributes:

Review

Quoted 8 lines

+``pcie_physical_function`` and ``pcie_bar_mapping``. These

(quoted diff) +7 -0

+identify which PCIe physical function and which BAR space in 
+that physical function the subsystem resides. XRT management 
+driver uses this information to bind *platform drivers* to the 
+subsystem instantiations. The platform drivers are found in 
+**xrt-lib.ko** kernel module defined later. Below is an example 
+of device tree for Alveo U50
+platform::

I might be missing something, but couldn't you structure the addressable endpoints in a way that encode the physical function as a parent / child relation?

Quoted 6 lines

Quoted 2 lines

Alveo driver does not generate the metadata. The metadata is formatted

and generated by HW tools when the Alveo HW platform is built.

Sure, but you control the tools that generate the metadata :) Your userland can structure / process it however it wants / needs?

XRT is a runtime software stack, it is not responsible for generating HW metadata. It is one of the consumers of these data. The shell design is generated by a sophisticated tool framework which is difficult to change.

However, we will take this as a feedback for future revision of the tool.

Thanks,
Max

Max,

Quoted 25 lines · Max Zhen

[...cut...]

Quoted 23 lines

Quoted 14 lines

Quoted 5 lines

Quoted 5 lines

(quoted diff) +5 -0

+xclbin over the User partition as part of DFX. When a user
+requests loading of a specific xclbin the xmgmt management
+driver reads the parent interface UUID specified in the xclbin
+and matches it with child interface UUID exported by Shell to
+determine if xclbin is compatible with the Shell. If match fails loading of xclbin is denied.

Quoted 8 lines

Review

Quoted 7 lines

(quoted diff) +7 -0

+
+xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl command.
+When loading xclbin xmgmt driver performs the following operations:
+
+1. Sanity check the xclbin contents 2. Isolate the User
+partition 3. Download the bitstream using the FPGA config engine (ICAP) 4.
+De-isolate the User partition

Is this modelled as bridges and regions?

Quoted 5 lines

Alveo drivers as written today do not use fpga bridge and region framework. It seems that if we add support for that framework, it’s possible to receive PR program request from kernel outside of xmgmt driver? Currently, we can’t support this and PR program can only be initiated using XRT’s runtime API in user space.

I'm not 100% sure I understand the concern here, let me reply to what I think I understand:

Quoted 8 lines

Quoted 6 lines

You're worried that if you use FPGA region as interface to accept PR requests something else could attempt to reconfigure the region from within the kernel using the FPGA Region API?

Assuming I got this right, I don't think this is a big deal. When you create the regions you control who gets the references to it.

Thanks for explaining. Yes, I think you got my point :-).

We can add code to make a region 'static' or 'one-time' or 'fixed'.

Quoted 10 lines

Quoted 4 lines

From what I've seen so far Regions seem to be roughly equivalent to Partitions, hence my surprise to see a new structure bypassing them.

I see where the gap is.

Regions in Linux is very different than "partitions" we have defined in xmgmt. Regions seem to be a software data structure representing an area on the FPGA that can be reprogrammed. This area is protected by the concept of "bridge" which can be disabled before program and reenabled after it. And you go through region when you need to reprogram this area.

Your central management driver can create / destroy regions at will. It can keep them in a list, array or tree.

Regions can but don't have to have bridges.

If you need to go through the central driver to reprogram a region, you can use that to figure out which region to program.

Quoted 4 lines

The "partition" is part of the main infrastructure of xmgmt driver, which represents a group of subdev drivers for each individual IP (HW subcomponents). Basically, xmgmt root driver is parent of several partitions who is, in turn, the parent of several subdev drivers. The parent manages the life cycle of its children here.

I don't see how this is conceptually different from what DFL does, and they managed to use Regions and Bridges.

If things are missing in the framework, please add them instead of rewriting an entire parallel framework.

Quoted 3 lines

We do have a partition to represent the group of subdevs/IPs in the reprogrammable area. And we also have partitions representing other areas which cannot be reprogrammed. So, it is difficult to use "Region" to implement "partition".

You implement your regions callbacks, you can return -EINVAL / -ENOTTY if you want to fail a reprogramming request to a static partion / region.

Quoted 4 lines

From what you have explained, it seems that even if I use region / bridge in xmgmt, we can still keep it private to xmgmt instead of exposing the interface to outside world, which we can't support anyway? This means that region will be used as an internal data structure for xmgmt. Since we can't simply replace partition with region, we might as well just use partition throughout the driver, instead of introducing two data structures and use them both in different places.

Think about your partition as an extension to a region that implements what you need to do for your case of enumerating and reprogramming that particular piece of your chip.

Quoted 2 lines

However, if using region/bridge can bring in other benefits, please let us know and we could see if we can also add this to xmgmt.

As maintainer I can say it brings the benefit of looking like existing infrastructure we have. We can add features to the framework as needed but blanket replacing the entire thing is always a hard sell.

Quoted 35 lines

Quoted 5 lines

Quoted 5 lines

Or maybe we have missed some points about the use case for this framework?

[...cut...]

Quoted 27 lines

Quoted 26 lines

Quoted 14 lines

Quoted 14 lines

+-----------------

(quoted diff) +3 -0

+
+As mentioned previously xsabin stores metadata which advertise
+HW

subsystems present in a partition.

(quoted diff) +3 -0

+The metadata is stored in device tree format with well defined
+schema. Subsystem instantiations are captured as children of
+``addressable_endpoints`` node. Subsystem nodes have standard

attributes like ``reg``, ``interrupts`` etc. Additionally the nodes also have PCIe specific attributes:
+``pcie_physical_function`` and ``pcie_bar_mapping``. These

(quoted diff) +3 -0

+identify which PCIe physical function and which BAR space in
+that physical function the subsystem resides. XRT management
+driver uses this information to bind *platform drivers* to the

Quoted 8 lines

Review

Quoted 4 lines

(quoted diff) +4 -0

+subsystem instantiations. The platform drivers are found in
+**xrt-lib.ko** kernel module defined later. Below is an example
+of device tree for Alveo U50
+platform::

I might be missing something, but couldn't you structure the addressable endpoints in a way that encode the physical function as a parent / child relation?

Alveo driver does not generate the metadata. The metadata is formatted
and generated by HW tools when the Alveo HW platform is built.

Quoted 5 lines

Quoted 2 lines

Sure, but you control the tools that generate the metadata :) Your userland can structure / process it however it wants / needs?

XRT is a runtime software stack, it is not responsible for generating HW metadata. It is one of the consumers of these data. The shell design is generated by a sophisticated tool framework which is difficult to change.

The Kernel userspace ABI is not going to change once it is merged, which is why we need to get it right. You can change your userspace code long time after it is merged into the kernel. The otherway round does not work.

If you're going to do device-tree you'll need device-tree maintainers to be ok with your bindings.

Quoted 4 lines

However, we will take this as a feedback for future revision of the tool.

Thanks,
Max

Btw: Can you fix your line-breaks :)

- Moritz

Hi Moritz,

I manually fixed some line breaks. Not sure why outlook is not doing it properly. Let me know if it still looks bad to you.

Please see my reply below.

Quoted 54 lines

Max,

Quoted 25 lines · Max Zhen

[...cut...]

Quoted 23 lines

Quoted 14 lines

Quoted 5 lines

Quoted 5 lines

(quoted diff) +5 -0

+xclbin over the User partition as part of DFX. When a user
+requests loading of a specific xclbin the xmgmt management
+driver reads the parent interface UUID specified in the xclbin
+and matches it with child interface UUID exported by Shell to
+determine if xclbin is compatible with the Shell. If match fails loading of xclbin is denied.

Quoted 8 lines

Review

Quoted 7 lines

(quoted diff) +7 -0

+
+xclbin loading is requested using ICAP_DOWNLOAD_AXLF ioctl command.
+When loading xclbin xmgmt driver performs the following operations:
+
+1. Sanity check the xclbin contents 2. Isolate the User
+partition 3. Download the bitstream using the FPGA config engine (ICAP) 4.
+De-isolate the User partition

Is this modelled as bridges and regions?

Quoted 5 lines

Alveo drivers as written today do not use fpga bridge and region framework. It seems that if we add support for that framework, it’s possible to receive PR program request from kernel outside of xmgmt driver? Currently, we can’t support this and PR program can only be initiated using XRT’s runtime API in user space.

I'm not 100% sure I understand the concern here, let me reply to what I think I understand:

Quoted 8 lines

Quoted 6 lines

You're worried that if you use FPGA region as interface to accept PR requests something else could attempt to reconfigure the region from within the kernel using the FPGA Region API?

Assuming I got this right, I don't think this is a big deal. When you create the regions you control who gets the references to it.

Thanks for explaining. Yes, I think you got my point :-).

We can add code to make a region 'static' or 'one-time' or 'fixed'.

Quoted 10 lines

Quoted 4 lines

From what I've seen so far Regions seem to be roughly equivalent to Partitions, hence my surprise to see a new structure bypassing them.

I see where the gap is.

Regions in Linux is very different than "partitions" we have defined in xmgmt. Regions seem to be a software data structure representing an area on the FPGA that can be reprogrammed. This area is protected by the concept of "bridge" which can be disabled before program and reenabled after it. And you go through region when you need to reprogram this area.

Your central management driver can create / destroy regions at will. It can keep them in a list, array or tree.

Quoted 4 lines

Regions can but don't have to have bridges.

If you need to go through the central driver to reprogram a region, you can use that to figure out which region to program.

That sounds fine. I can create a region and call into it from xmgmt for PR programing. The region will, then, call the xmgmt's fpga manager to program it.

Quoted 20 lines

Quoted 5 lines

The "partition" is part of the main infrastructure of xmgmt driver, which represents a group of subdev drivers for each individual IP (HW subcomponents). Basically, xmgmt root driver is parent of several partitions who is, in turn, the parent of several subdev drivers. The parent manages the life cycle of its children here.

I don't see how this is conceptually different from what DFL does, and they managed to use Regions and Bridges.

If things are missing in the framework, please add them instead of rewriting an entire parallel framework.

Quoted 3 lines

We do have a partition to represent the group of subdevs/IPs in the reprogrammable area. And we also have partitions representing other areas which cannot be reprogrammed. So, it is difficult to use "Region" to implement "partition".

You implement your regions callbacks, you can return -EINVAL / -ENOTTY if you want to fail a reprogramming request to a static partion / region.

Quoted 8 lines

Quoted 4 lines

From what you have explained, it seems that even if I use region / bridge in xmgmt, we can still keep it private to xmgmt instead of exposing the interface to outside world, which we can't support anyway? This means that region will be used as an internal data structure for xmgmt. Since we can't simply replace partition with region, we might as well just use partition throughout the driver, instead of introducing two data structures and use them both in different places.

Think about your partition as an extension to a region that implements what you need to do for your case of enumerating and reprogramming that particular piece of your chip.

Yes, we can add region / bridges to represent the PR area and use it in our code path for reprogramming the PR area. I think what we will do is to instantiate a region instance for the PR area and associate it with the FPGA manager in xmgmt for reprogramming it. We can also instantiate bridges and map the "ULP gate" subdev driver to it in xmgmt. Thus, we could incorporate region and bridge data structures in xmgmt for PR reprogramming.

This will be a non-trivial change for us. I'd like to confirm that this is what you are looking for before we start working on the change. Let us know :-).

From: Sonal Santan <[email protected]>

Add XRT UAPI header files which describe flash layout, XRT mailbox protocol, xclBin/axlf FPGA image container format and XRT management physical function driver ioctl interfaces.

flash_xrt_data.h:
Layout used by XRT to store private data on flash.

mailbox_proto.h:
Mailbox opcodes and high level data structures representing various kinds of information like sensors, clock, etc.

mailbox_transport.h:
Transport protocol used by mailbox.

xclbin.h:
Container format used to store compiled FPGA image which includes bitstream and metadata.

-----Original Message-----
From: Moritz Fischer <[email protected]>
Sent: Monday, November 30, 2020 8:27 PM
To: Sonal Santan <[email protected]>
Cc: [email protected]; [email protected]; Max Zhen <[email protected]>; Lizhi Hou <[email protected]>; Michal Simek <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH Xilinx Alveo 2/8] fpga: xrt: Add UAPI header files

Hi Sonal,

Quoted 13 lines · Sonal Santan

From: Sonal Santan <[email protected]>

Add XRT UAPI header files which describe flash layout, XRT mailbox protocol, xclBin/axlf FPGA image container format and XRT management physical function driver ioctl interfaces.

flash_xrt_data.h:
Layout used by XRT to store private data on flash.

mailbox_proto.h:
Mailbox opcodes and high level data structures representing various kinds of information like sensors, clock, etc.

Quoted 6 lines

mailbox_transport.h:
Transport protocol used by mailbox.

xclbin.h:
Container format used to store compiled FPGA image which includes bitstream and metadata.

Can these headers be introduced together with the code that uses them as logical change?

I haven't looked too closely, but it helps reviewing if you can break it into smaller pieces that can stand by themselves.

Thanks,
Moritz

Hi Moritz,

Quoted 33 lines

-----Original Message-----
From: Moritz Fischer <[email protected]>
Sent: Monday, November 30, 2020 8:27 PM
To: Sonal Santan <[email protected]>
Cc: [email protected]; [email protected]; Max Zhen <[email protected]>; Lizhi Hou <[email protected]>; Michal Simek <[email protected]>; Stefano Stabellini <[email protected]>; [email protected]
Subject: Re: [PATCH Xilinx Alveo 2/8] fpga: xrt: Add UAPI header files

Hi Sonal,

Quoted 13 lines · Sonal Santan

From: Sonal Santan <[email protected]>

Add XRT UAPI header files which describe flash layout, XRT mailbox protocol, xclBin/axlf FPGA image container format and XRT management physical function driver ioctl interfaces.

flash_xrt_data.h:
Layout used by XRT to store private data on flash.

mailbox_proto.h:
Mailbox opcodes and high level data structures representing various kinds of information like sensors, clock, etc.

Quoted 6 lines

mailbox_transport.h:
Transport protocol used by mailbox.

xclbin.h:
Container format used to store compiled FPGA image which includes bitstream and metadata.

Quoted 6 lines

Can these headers be introduced together with the code that uses them as logical change?

I haven't looked too closely, but it helps reviewing if you can break it into smaller pieces that can stand by themselves.

These UAPI header files are used by multiple source files hence I wanted to get these reviewed separately. However if this is getting in the way, in the next version of the patch series I would look into arranging the files differently.

You can browse the changes here as well--
https://github.com/Xilinx/linux-xoclv2/tree/xrtv2-A