Enable Drivers for Intel MIC X100 Coprocessors.

PATCH Superseded Sudeep Dutt 2 participants 37 messages Aug 21, 2013 – Sep 4, 2013

Subsystems (3)

Documentation Char and misc drivers virtio core

lore

Share & tools

lore

Sudeep Dutt

author

2013-08-21 17:50 UTC #0 lore raw

v3 cover Enable Drivers for Intel MIC X100 Coprocessors.

ChangeLog:
=========

v2 => v3:
a) Patch 1 data structure cleanups, header file include cleanups, IDA interface reuse and switching to device_create_with_groups(..) as per feedback from Greg Kroah-Hartman.
b) Patch 7 signal documentation, sleep workaround removal and sysfs access API cleanups as per feedback from Michael S. Tsirkin.

v1 => v2: @ http://lwn.net/Articles/563131/
a) License wording cleanup, sysfs ABI documentation, patch 1 refactoring into 3 smaller patches and function renames, as per feedback from Greg Kroah-Hartman.
b) Use VRINGH infrastructure for accessing virtio rings from the host in patch 5, as per feedback from Michael S. Tsirkin.

v1: Initial post @ https://lwn.net/Articles/561314/

Description:
============

An Intel MIC X100 device is a PCIe form factor add-in coprocessor card based on the Intel Many Integrated Core (MIC) architecture that runs a Linux OS. It is a PCIe endpoint in a platform and therefore implements the three required standard address spaces i.e. configuration, memory and I/O. The host OS loads a device driver as is typical for PCIe devices. The card itself runs a bootstrap after reset that transfers control to the card OS downloaded from the host driver. The card OS as shipped by Intel is a Linux kernel with modifications for the X100 devices.

Since it is a PCIe card, it does not have the ability to host hardware devices for networking, storage and console. We provide these devices on X100 coprocessors thus enabling a self-bootable equivalent environment for applications. A key benefit of our solution is that it leverages the standard virtio framework for network, disk and console devices, though in our case the virtio framework is used across a PCIe bus.

Here is a block diagram of the various components described above. The virtio backends are situated on the host rather than the card given better single threaded performance for the host compared to MIC, the ability of the host to initiate DMA's to/from the card using the MIC DMA engine and the fact that the virtio block storage backend can only be on the host.

       +----------+           |             +----------+
       | Card OS  |           |             | Host OS  |
       +----------+           |             +----------+
                              |
+-------+ +--------+ +------+ | +---------+  +--------+ +--------+
| Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
| Net   | |Console | |Block | | |Net      |  |Console | |Block   |
| Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
+-------+ +--------+ +------+ | +---------+  +--------+ +--------+
    |         |         |     |      |            |         |
    |         |         |     |User  |            |         |
    |         |         |     |------|------------|---------|-------
    +-------------------+     |Kernel +--------------------------+
              |               |       | Virtio over PCIe IOCTLs  |
              |               |       +--------------------------+
      +--------------+        |                   |
      |Intel MIC     |        |            +---------------+
      |Card Driver   |        |            |Intel MIC      |
      +--------------+        |            |Host Driver    |
              |               |            +---------------+
              |               |                   |
     +-------------------------------------------------------------+
     |                                                             |
     |                    PCIe Bus                                 |
     +-------------------------------------------------------------+

The following series of patches are partitioned as follows:

Patch 1: This patch introduces the "Intel MIC Host Driver" in the block diagram which does the following:
a) Initializes the Intel MIC X100 PCIe devices.
b) Provides sysfs entries for family and stepping information.

Patch 2: This patch enables the following features in the "Intel MIC Host Driver" in the block diagram:
a) MSIx, MSI and legacy interrupt support.
b) System Memory Page Table(SMPT) support. SMPT enables system memory access from the card. On X100 devices the host can program 32 SMPT registers each capable of accessing 16GB of system memory address space from X100 devices. The registers can thereby be used to access a cumulative 512GB of system memory address space from X100 devices at any point in time.

Patch 3: This patch enables the following features in the "Intel MIC Host Driver" in the block diagram:
a) Boots and shuts down the card via sysfs entries.
b) Allocates and maps a device page for communication with the card driver and updates the device page address via scratchpad registers.
c) Provides sysfs entries for shutdown status, kernel command line, firmware, ramdisk, bootmode and log buffer information.

Patch 4: This patch introduces the "Intel MIC Card Driver" in the block diagram which does the following:
a) Initializes the Intel MIC X100 platform device and driver.
b) Sets up support to handle shutdown requests from the host.
c) Maps the device page after obtaining the device page address from the scratchpad registers updated by the host.
d) Informs the host upon a card crash by registering a panic notifier.
e) Informs the host upon a poweroff/halt event.

Patch 5: This patch introduces the host "Virtio over PCIe" interface for Intel MIC. It allows creating user space backends on the host and instantiating virtio devices for them on the Intel MIC card. It uses the existing VRINGH infrastructure in the kernel to access virtio rings from the host. A character device per MIC is exposed with IOCTL, mmap and poll callbacks. This allows the user space backend to:
(a) add/remove a virtio device via a device page.
(b) map (R/O) virtio rings and device page to user space.
(c) poll for availability of data.
(d) copy a descriptor or entire descriptor chain to/from the card.
(e) modify virtio configuration.
(f) handle virtio device reset.
The buffers are copied over using CPU copies for this initial patch and host initiated MIC DMA support is planned for future patches. The avail and desc virtio rings are in host memory and the used ring is in card memory to maximize writes across PCIe for performance.

Patch 6: This patch introduces the card "Virtio over PCIe" interface for Intel MIC. It allows virtio drivers on the card to communicate with their user space backends on the host via a device page. Ring 3 apps on the host can add, remove and configure virtio devices. A thin MIC specific virtio_config_ops is implemented which is borrowed heavily from previous similar implementations in lguest and s390 @ drivers/lguest/lguest_device.c
drivers/s390/kvm/kvm_virtio.c

Patch 7: This patch introduces a sample user space daemon which implements the virtio device backends on the host. The daemon creates/removes/configures virtio device backends by communicating with the Intel MIC Host Driver. The virtio devices currently supported are virtio net, virtio console and virtio block. Virtio net supports TSO/GSO. The daemon also monitors card shutdown status and takes appropriate actions like killing the virtio backends and resetting the card upon card shutdown and crashes.

The patches have been compiled/validated against v3.11-rc5.

Ashutosh Dixit (2):
Intel MIC Host Driver Changes for Virtio Devices. Intel MIC Card Driver Changes for Virtio Devices.

Caz Yokoyama (1):
Sample Implementation of Intel MIC User Space Daemon.

Dasaratharaman Chandramouli (1):
Intel MIC Host Driver Interrupt/SMPT support.

Sudeep Dutt (3):
Intel MIC Host Driver for X100 family. Intel MIC Host Driver, card OS state management. Intel MIC Card Driver for X100 family.

 Documentation/ABI/testing/sysfs-class-mic.txt |  147 +++
 Documentation/mic/mic_overview.txt            |   49 +
 Documentation/mic/mpssd/.gitignore            |    1 +
 Documentation/mic/mpssd/Makefile              |   19 +
 Documentation/mic/mpssd/micctrl               |  173 +++
 Documentation/mic/mpssd/mpss                  |  202 +++
 Documentation/mic/mpssd/mpssd.c               | 1701 +++++++++++++++++++++++++
 Documentation/mic/mpssd/mpssd.h               |  100 ++
 Documentation/mic/mpssd/sysfs.c               |  102 ++
 drivers/misc/Kconfig                          |    1 +
 drivers/misc/Makefile                         |    1 +
 drivers/misc/mic/Kconfig                      |   39 +
 drivers/misc/mic/Makefile                     |    6 +
 drivers/misc/mic/card/Makefile                |   11 +
 drivers/misc/mic/card/mic_debugfs.c           |  130 ++
 drivers/misc/mic/card/mic_device.c            |  306 +++++
 drivers/misc/mic/card/mic_device.h            |  133 ++
 drivers/misc/mic/card/mic_virtio.c            |  631 +++++++++
 drivers/misc/mic/card/mic_virtio.h            |   77 ++
 drivers/misc/mic/card/mic_x100.c              |  256 ++++
 drivers/misc/mic/card/mic_x100.h              |   48 +
 drivers/misc/mic/common/mic_device.h          |   51 +
 drivers/misc/mic/host/Makefile                |   13 +
 drivers/misc/mic/host/mic_boot.c              |  185 +++
 drivers/misc/mic/host/mic_debugfs.c           |  496 +++++++
 drivers/misc/mic/host/mic_device.h            |  300 +++++
 drivers/misc/mic/host/mic_fops.c              |  221 ++++
 drivers/misc/mic/host/mic_fops.h              |   32 +
 drivers/misc/mic/host/mic_main.c              | 1098 ++++++++++++++++
 drivers/misc/mic/host/mic_smpt.c              |  440 +++++++
 drivers/misc/mic/host/mic_smpt.h              |   98 ++
 drivers/misc/mic/host/mic_sysfs.c             |  468 +++++++
 drivers/misc/mic/host/mic_virtio.c            |  704 ++++++++++
 drivers/misc/mic/host/mic_virtio.h            |  138 ++
 drivers/misc/mic/host/mic_x100.c              |  573 +++++++++
 drivers/misc/mic/host/mic_x100.h              |   99 ++
 include/uapi/linux/Kbuild                     |    2 +
 include/uapi/linux/mic_common.h               |  238 ++++
 include/uapi/linux/mic_ioctl.h                |   74 ++
 39 files changed, 9363 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt
 create mode 100644 Documentation/mic/mic_overview.txt
 create mode 100644 Documentation/mic/mpssd/.gitignore
 create mode 100644 Documentation/mic/mpssd/Makefile
 create mode 100755 Documentation/mic/mpssd/micctrl
 create mode 100755 Documentation/mic/mpssd/mpss
 create mode 100644 Documentation/mic/mpssd/mpssd.c
 create mode 100644 Documentation/mic/mpssd/mpssd.h
 create mode 100644 Documentation/mic/mpssd/sysfs.c
 create mode 100644 drivers/misc/mic/Kconfig
 create mode 100644 drivers/misc/mic/Makefile
 create mode 100644 drivers/misc/mic/card/Makefile
 create mode 100644 drivers/misc/mic/card/mic_debugfs.c
 create mode 100644 drivers/misc/mic/card/mic_device.c
 create mode 100644 drivers/misc/mic/card/mic_device.h
 create mode 100644 drivers/misc/mic/card/mic_virtio.c
 create mode 100644 drivers/misc/mic/card/mic_virtio.h
 create mode 100644 drivers/misc/mic/card/mic_x100.c
 create mode 100644 drivers/misc/mic/card/mic_x100.h
 create mode 100644 drivers/misc/mic/common/mic_device.h
 create mode 100644 drivers/misc/mic/host/Makefile
 create mode 100644 drivers/misc/mic/host/mic_boot.c
 create mode 100644 drivers/misc/mic/host/mic_debugfs.c
 create mode 100644 drivers/misc/mic/host/mic_device.h
 create mode 100644 drivers/misc/mic/host/mic_fops.c
 create mode 100644 drivers/misc/mic/host/mic_fops.h
 create mode 100644 drivers/misc/mic/host/mic_main.c
 create mode 100644 drivers/misc/mic/host/mic_smpt.c
 create mode 100644 drivers/misc/mic/host/mic_smpt.h
 create mode 100644 drivers/misc/mic/host/mic_sysfs.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.h
 create mode 100644 drivers/misc/mic/host/mic_x100.c
 create mode 100644 drivers/misc/mic/host/mic_x100.h
 create mode 100644 include/uapi/linux/mic_common.h
 create mode 100644 include/uapi/linux/mic_ioctl.h

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:50 UTC #1 lore raw

PATCH v3 1/7 Intel MIC Host Driver for X100 family.

This patch enables the following:
a) Initializes the Intel MIC X100 PCIe devices.
b) Provides sysfs entries for family and stepping information.

Co-author: Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 Documentation/ABI/testing/sysfs-class-mic.txt |  34 +++
 drivers/misc/Kconfig                          |   1 +
 drivers/misc/Makefile                         |   1 +
 drivers/misc/mic/Kconfig                      |  19 ++
 drivers/misc/mic/Makefile                     |   5 +
 drivers/misc/mic/common/mic_device.h          |  37 +++
 drivers/misc/mic/host/Makefile                |   8 +
 drivers/misc/mic/host/mic_device.h            | 109 +++++++++
 drivers/misc/mic/host/mic_main.c              | 309 ++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_sysfs.c             |  97 ++++++++
 drivers/misc/mic/host/mic_x100.c              |  75 +++++++
 drivers/misc/mic/host/mic_x100.h              |  47 ++++
 12 files changed, 742 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt
 create mode 100644 drivers/misc/mic/Kconfig
 create mode 100644 drivers/misc/mic/Makefile
 create mode 100644 drivers/misc/mic/common/mic_device.h
 create mode 100644 drivers/misc/mic/host/Makefile
 create mode 100644 drivers/misc/mic/host/mic_device.h
 create mode 100644 drivers/misc/mic/host/mic_main.c
 create mode 100644 drivers/misc/mic/host/mic_sysfs.c
 create mode 100644 drivers/misc/mic/host/mic_x100.c
 create mode 100644 drivers/misc/mic/host/mic_x100.h

Documentation/ABI/testing/sysfs-class-mic.txt new file +34 -0

new file mode 100644
index 0000000..09eb3c6
@@ -0,0 +1,34 @@
+What:		/sys/class/mic/
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The mic class directory belongs to Intel MIC devices and
+		provides information per MIC device. An Intel MIC device is a
+		PCIe form factor add-in Coprocessor card based on the Intel Many
+		Integrated Core (MIC) architecture that runs a Linux OS.
+
+What:		/sys/class/mic/mic(x)
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The directories /sys/class/mic/mic0, /sys/class/mic/mic1 etc.,
+		represent MIC devices (0,1,..etc). Each directory has
+		information specific to that MIC device.
+
+What:		/sys/class/mic/mic(x)/family
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the Coprocessor family for an Intel
+		MIC device. For example - "x100"
+
+What:		/sys/class/mic/mic(x)/stepping
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the silicon stepping for an Intel
+		MIC device. For example - "A0" or "B0"

drivers/misc/Kconfig +1 -0

index 8dacd4c..e760715 100644
@@ -537,4 +537,5 @@ source "drivers/misc/carma/Kconfig"
 source "drivers/misc/altera-stapl/Kconfig"
 source "drivers/misc/mei/Kconfig"
 source "drivers/misc/vmw_vmci/Kconfig"
+source "drivers/misc/mic/Kconfig"
 endmenu

drivers/misc/Makefile +1 -0

index c235d5b..0b7ea3e 100644
@@ -53,3 +53,4 @@ obj-$(CONFIG_INTEL_MEI)		+= mei/
 obj-$(CONFIG_VMWARE_VMCI)	+= vmw_vmci/
 obj-$(CONFIG_LATTICE_ECP3_CONFIG)	+= lattice-ecp3-config.o
 obj-$(CONFIG_SRAM)		+= sram.o
+obj-y				+= mic/

drivers/misc/mic/Kconfig new file +19 -0

new file mode 100644
index 0000000..aaefd0c
@@ -0,0 +1,19 @@
+comment "Intel MIC Host Driver"
+
+config INTEL_MIC_HOST
+	tristate "Intel MIC Host Driver"
+	depends on 64BIT && PCI
+	default N
+	help
+	  This enables Host Driver support for the Intel Many Integrated
+	  Core (MIC) family of PCIe form factor coprocessor devices that
+	  run a 64 bit Linux OS. The driver manages card OS state and
+	  enables communication between host and card. Intel MIC X100
+	  devices are currently supported.
+
+	  If you are building a host kernel with an Intel MIC device then
+	  say M (recommended) or Y, else say N. If unsure say N.
+
+	  More information about the Intel MIC family as well as the Linux
+	  OS and tools for MIC to use with this driver are available from
+	  <http://software.intel.com/en-us/mic-developer>.

drivers/misc/mic/Makefile new file +5 -0

new file mode 100644
index 0000000..8e72421
@@ -0,0 +1,5 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += host/

drivers/misc/mic/common/mic_device.h new file +37 -0

new file mode 100644
index 0000000..f02262e
@@ -0,0 +1,37 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC driver.
+ *
+ */
+#ifndef __MIC_COMMON_DEVICE_H_
+#define __MIC_COMMON_DEVICE_H_
+
+/**
+ * struct mic_mw - MIC memory window
+ *
+ * @pa: Base physical address.
+ * @va: Base ioremap'd virtual address.
+ * @len: Size of the memory window.
+ */
+struct mic_mw {
+	phys_addr_t pa;
+	void __iomem *va;
+	resource_size_t len;
+};
+
+#endif

drivers/misc/mic/host/Makefile new file +8 -0

new file mode 100644
index 0000000..93b9d25
@@ -0,0 +1,8 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += mic_host.o
+mic_host-objs := mic_main.o
+mic_host-objs += mic_x100.o
+mic_host-objs += mic_sysfs.o

drivers/misc/mic/host/mic_device.h new file +109 -0

109 lines not on this page · raw message

drivers/misc/mic/host/mic_main.c new file +309 -0

309 lines not on this page · raw message

drivers/misc/mic/host/mic_sysfs.c new file +97 -0

97 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.c new file +75 -0

75 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.h new file +47 -0

47 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:50 UTC #10 lore raw

PATCH v3 2/7 Intel MIC Host Driver Interrupt/SMPT support.

In reply to Sudeep Dutt

From: Dasaratharaman Chandramouli <[email protected]>

This patch enables the following features:
a) MSIx, MSI and legacy interrupt support.
b) System Memory Page Table(SMPT) support. SMPT enables system memory access from the card. On X100 devices the host can program 32 SMPT registers each capable of accessing 16GB of system memory address space from X100 devices. The registers can thereby be used to access a cumulative 512GB of system memory address space from X100 devices at any point in time.

Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 drivers/misc/mic/host/Makefile     |   1 +
 drivers/misc/mic/host/mic_device.h | 126 ++++++++
 drivers/misc/mic/host/mic_main.c   | 647 ++++++++++++++++++++++++++++++++++++-
 drivers/misc/mic/host/mic_smpt.c   | 440 +++++++++++++++++++++++++
 drivers/misc/mic/host/mic_smpt.h   |  98 ++++++
 drivers/misc/mic/host/mic_x100.c   | 247 ++++++++++++++
 drivers/misc/mic/host/mic_x100.h   |  40 +++
 7 files changed, 1597 insertions(+), 2 deletions(-)
 create mode 100644 drivers/misc/mic/host/mic_smpt.c
 create mode 100644 drivers/misc/mic/host/mic_smpt.h

drivers/misc/mic/host/Makefile +1 -0

index 93b9d25..3702d2a 100644
@@ -6,3 +6,4 @@ obj-$(CONFIG_INTEL_MIC_HOST) += mic_host.o
 mic_host-objs := mic_main.o
 mic_host-objs += mic_x100.o
 mic_host-objs += mic_sysfs.o
+mic_host-objs += mic_smpt.o

drivers/misc/mic/host/mic_device.h +126 -0

169 lines not on this page · raw message

drivers/misc/mic/host/mic_main.c +645 -2

697 lines not on this page · raw message

drivers/misc/mic/host/mic_smpt.c new file +440 -0

440 lines not on this page · raw message

drivers/misc/mic/host/mic_smpt.h new file +98 -0

98 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.c +247 -0

262 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.h +40 -0

index 1f4e630..fd98b2b 100644
@@ -42,6 +42,46 @@
 
 #define MIC_X100_SBOX_BASE_ADDRESS 0x00010000
 #define MIC_X100_SBOX_SPAD0 0x0000AB20
+#define MIC_X100_SBOX_SICR0_DBR(x) ((x) & 0xf)
+#define MIC_X100_SBOX_SICR0_DMA(x) (((x) >> 8) & 0xff)
+#define MIC_X100_SBOX_SICE0_DBR(x) ((x) & 0xf)
+#define MIC_X100_SBOX_DBR_BITS(x) ((x) & 0xf)
+#define MIC_X100_SBOX_SICE0_DMA(x) (((x) >> 8) & 0xff)
+#define MIC_X100_SBOX_DMA_BITS(x) (((x) & 0xff) << 8)
+
+#define MIC_X100_SBOX_APICICR0 0x0000A9D0
+#define MIC_X100_SBOX_SICR0 0x00009004
+#define MIC_X100_SBOX_SICE0 0x0000900C
+#define MIC_X100_SBOX_SICC0 0x00009010
+#define MIC_X100_SBOX_SIAC0 0x00009014
+#define MIC_X100_SBOX_MSIXPBACR 0x00009084
+#define MIC_X100_SBOX_MXAR0 0x00009044
+#define MIC_X100_SBOX_SMPT00 0x00003100
+#define MIC_X100_SBOX_RDMASR0 0x0000B180
+
+#define MIC_X100_DOORBELL_IDX_START 0
+#define MIC_X100_NUM_DOORBELL 4
+#define MIC_X100_DMA_IDX_START 8
+#define MIC_X100_NUM_DMA 8
+#define MIC_X100_ERR_IDX_START 30
+#define MIC_X100_NUM_ERR 1
+
+#define MIC_X100_NUM_SBOX_IRQ 8
+#define MIC_X100_NUM_RDMASR_IRQ 8
+#define MIC_X100_RDMASR_IRQ_BASE 17
+#define MIC_NUM_OFFSETS 32
+
+static const u16 mic_x100_intr_init[] = {
+		MIC_X100_DOORBELL_IDX_START,
+		MIC_X100_DMA_IDX_START,
+		MIC_X100_ERR_IDX_START,
+		MIC_X100_NUM_DOORBELL,
+		MIC_X100_NUM_DMA,
+		MIC_X100_NUM_ERR,
+};
+
 extern struct mic_hw_ops mic_x100_ops;
+extern struct mic_smpt_ops mic_x100_smpt_ops;
+extern struct mic_hw_intr_ops mic_x100_intr_ops;
 
 #endif

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:50 UTC #11 lore raw

PATCH v3 3/7 Intel MIC Host Driver, card OS state management.

In reply to Sudeep Dutt

This patch enables the following features:
a) Boots and shuts down the card via sysfs entries.
b) Allocates and maps a device page for communication with the card driver and updates the device page address via scratchpad registers.
c) Provides sysfs entries for shutdown status, kernel command line, ramdisk and log buffer information.

Co-author: Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 Documentation/ABI/testing/sysfs-class-mic.txt | 113 ++++++++
 drivers/misc/mic/common/mic_device.h          |   7 +
 drivers/misc/mic/host/Makefile                |   2 +
 drivers/misc/mic/host/mic_boot.c              | 184 +++++++++++++
 drivers/misc/mic/host/mic_debugfs.c           | 356 ++++++++++++++++++++++++
 drivers/misc/mic/host/mic_device.h            |  60 +++++
 drivers/misc/mic/host/mic_main.c              | 129 ++++++++-
 drivers/misc/mic/host/mic_sysfs.c             | 371 ++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_x100.c              | 251 +++++++++++++++++
 drivers/misc/mic/host/mic_x100.h              |  12 +
 include/uapi/linux/Kbuild                     |   1 +
 include/uapi/linux/mic_common.h               |  74 +++++
 12 files changed, 1556 insertions(+), 4 deletions(-)
 create mode 100644 drivers/misc/mic/host/mic_boot.c
 create mode 100644 drivers/misc/mic/host/mic_debugfs.c
 create mode 100644 include/uapi/linux/mic_common.h

Documentation/ABI/testing/sysfs-class-mic.txt +113 -0

index 09eb3c6..82cdad3 100644
@@ -32,3 +32,116 @@ Contact:	Sudeep Dutt <[email protected]>
 Description:
 		Provides information about the silicon stepping for an Intel
 		MIC device. For example - "A0" or "B0"
+
+What:		/sys/class/mic/mic(x)/state
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		When read, this entry provides the current state of an Intel
+		MIC device in the context of the card OS. Possible values that
+		will be read are:
+		"offline" - The MIC device is ready to boot the card OS.
+		"online" - The MIC device has initiated booting a card OS.
+		"shutting_down" - The card OS is shutting down.
+		"reset_failed" - The MIC device has failed to reset.
+
+		When written, this sysfs entry triggers different state change
+		operations depending upon the current state of the card OS.
+		Acceptable values are:
+		"boot" - Boot the card OS image specified by the combination
+			 of firmware, ramdisk, cmdline and bootmode
+			sysfs entries.
+		"reset" - Initiates device reset.
+		"shutdown" - Initiates card OS shutdown.
+
+What:		/sys/class/mic/mic(x)/shutdown_status
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		An Intel MIC device runs a Linux OS during its operation. This
+		OS can shutdown because of various reasons. When read, this
+		entry provides the status on why the card OS was shutdown.
+		Possible values are:
+		"nop" -  shutdown status is not applicable, when the card OS is
+			"online"
+		"crashed" - Shutdown because of a HW or SW crash.
+		"halted" - Shutdown because of a halt command.
+		"poweroff" - Shutdown because of a poweroff command.
+		"restart" - Shutdown because of a restart command.
+
+What:		/sys/class/mic/mic(x)/cmdline
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		An Intel MIC device runs a Linux OS during its operation. Before
+		booting this card OS, it is possible to pass kernel command line
+		options to configure various features in it, similar to
+		self-bootable machines. When read, this entry provides
+		information about the current kernel command line options set to
+		boot the card OS. This entry can be written to change the
+		existing kernel command line options. Typically, the user would
+		want to read the current command line options, append new ones
+		or modify existing ones and then write the whole kernel command
+		line back to this entry.
+
+What:		/sys/class/mic/mic(x)/firmware
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		When read, this sysfs entry provides the path name under
+		/lib/firmware/ where the firmware image to be booted on the
+		card can be found. The entry can be written to change the
+		firmware image location under /lib/firmware/.
+
+What:		/sys/class/mic/mic(x)/ramdisk
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		When read, this sysfs entry provides the path name under
+		/lib/firmware/ where the ramdisk image to be used during card
+		OS boot can be found. The entry can be written to change
+		the ramdisk image location under /lib/firmware/.
+
+What:		/sys/class/mic/mic(x)/bootmode
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		When read, this sysfs entry provides the current bootmode for
+		the card. This sysfs entry can be written with the following
+		valid strings:
+		a) linux - Boot a Linux image.
+		b) elf - Boot an elf image for flash updates.
+
+What:		/sys/class/mic/mic(x)/log_buf_addr
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		An Intel MIC device runs a Linux OS during its operation. For
+		debugging purpose and early kernel boot messages, the user can
+		access the card OS log buffer via debugfs. When read, this entry
+		provides the kernel virtual address of the buffer where the card
+		OS log buffer can be read. This entry is written by the host
+		configuration daemon to set the log buffer address. The correct
+		log buffer address to be written can be found in the System.map
+		file of the card OS.
+
+What:		/sys/class/mic/mic(x)/log_buf_len
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		An Intel MIC device runs a Linux OS during its operation. For
+		debugging purpose and early kernel boot messages, the user can
+		access the card OS log buffer via debugfs. When read, this entry
+		provides the kernel virtual address where the card OS log buffer
+		length can be read. This entry is written by host configuration
+		daemon to set the log buffer length address. The correct log
+		buffer length address to be written can be found in the
+		System.map file of the card OS.

drivers/misc/mic/common/mic_device.h +7 -0

index f02262e..6440e9d 100644
@@ -34,4 +34,11 @@ struct mic_mw {
 	resource_size_t len;
 };
 
+/*
+ * Scratch pad register offsets used by the host to communicate
+ * device page DMA address to the card.
+ */
+#define MIC_DPLO_SPAD 14
+#define MIC_DPHI_SPAD 15
+
 #endif

drivers/misc/mic/host/Makefile +2 -0

index 3702d2a..98bf565 100644
@@ -7,3 +7,5 @@ mic_host-objs := mic_main.o
 mic_host-objs += mic_x100.o
 mic_host-objs += mic_sysfs.o
 mic_host-objs += mic_smpt.o
+mic_host-objs += mic_boot.o
+mic_host-objs += mic_debugfs.o

drivers/misc/mic/host/mic_boot.c new file +184 -0

184 lines not on this page · raw message

drivers/misc/mic/host/mic_debugfs.c new file +356 -0

356 lines not on this page · raw message

drivers/misc/mic/host/mic_device.h +60 -0

88 lines not on this page · raw message

drivers/misc/mic/host/mic_main.c +125 -4

195 lines not on this page · raw message

drivers/misc/mic/host/mic_sysfs.c +371 -0

389 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.c +251 -0

269 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.h +12 -0

index fd98b2b..227f73b 100644
@@ -70,6 +70,15 @@
 #define MIC_X100_NUM_RDMASR_IRQ 8
 #define MIC_X100_RDMASR_IRQ_BASE 17
 #define MIC_NUM_OFFSETS 32
+#define MIC_X100_SPAD2_DOWNLOAD_STATUS(x) ((x) & 0x1)
+#define MIC_X100_SPAD2_APIC_ID(x)	(((x) >> 1) & 0x1ff)
+#define MIC_X100_SPAD2_DOWNLOAD_ADDR(x) ((x) & 0xfffff000)
+#define MIC_X100_SBOX_APICICR7 0x0000AA08
+#define MIC_X100_SBOX_RGCR 0x00004010
+#define MIC_X100_SBOX_SDBIC0 0x0000CC90
+#define MIC_X100_DOWNLOAD_INFO 2
+#define MIC_X100_FW_SIZE 5
+#define MIC_X100_POSTCODE 0x242c
 
 static const u16 mic_x100_intr_init[] = {
 		MIC_X100_DOORBELL_IDX_START,

@@ -80,6 +89,9 @@ static const u16 mic_x100_intr_init[] = {
 		MIC_X100_NUM_ERR,
 };
 
+/* Host->Card(bootstrap) Interrupt Vector */
+#define MIC_X100_BSP_INTERRUPT_VECTOR 229
+
 extern struct mic_hw_ops mic_x100_ops;
 extern struct mic_smpt_ops mic_x100_smpt_ops;
 extern struct mic_hw_intr_ops mic_x100_intr_ops;

include/uapi/linux/Kbuild +1 -0

index 997f9f2..e3f7b37 100644
@@ -240,6 +240,7 @@ header-y += media.h
 header-y += mei.h
 header-y += mempolicy.h
 header-y += meye.h
+header-y += mic_common.h
 header-y += mii.h
 header-y += minix_fs.h
 header-y += mman.h

include/uapi/linux/mic_common.h new file +74 -0

74 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:50 UTC #12 lore raw

PATCH v3 4/7 Intel MIC Card Driver for X100 family.

In reply to Sudeep Dutt

This patch does the following:
a) Initializes the Intel MIC X100 platform device and driver.
b) Sets up support to handle shutdown requests from the host.
c) Maps the device page after obtaining the device page address from the scratchpad registers updated by the host.
d) Informs the host upon a card crash by registering a panic notifier.
e) Informs the host upon a poweroff/halt event.

Co-author: Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 drivers/misc/mic/Kconfig            |  18 +++
 drivers/misc/mic/Makefile           |   1 +
 drivers/misc/mic/card/Makefile      |  10 ++
 drivers/misc/mic/card/mic_debugfs.c | 130 ++++++++++++++++
 drivers/misc/mic/card/mic_device.c  | 299 ++++++++++++++++++++++++++++++++++++
 drivers/misc/mic/card/mic_device.h  | 133 ++++++++++++++++
 drivers/misc/mic/card/mic_x100.c    | 256 ++++++++++++++++++++++++++++++
 drivers/misc/mic/card/mic_x100.h    |  48 ++++++
 8 files changed, 895 insertions(+)
 create mode 100644 drivers/misc/mic/card/Makefile
 create mode 100644 drivers/misc/mic/card/mic_debugfs.c
 create mode 100644 drivers/misc/mic/card/mic_device.c
 create mode 100644 drivers/misc/mic/card/mic_device.h
 create mode 100644 drivers/misc/mic/card/mic_x100.c
 create mode 100644 drivers/misc/mic/card/mic_x100.h

drivers/misc/mic/Kconfig +18 -0

index aaefd0c..279a2e6 100644
@@ -17,3 +17,21 @@ config INTEL_MIC_HOST
 	  More information about the Intel MIC family as well as the Linux
 	  OS and tools for MIC to use with this driver are available from
 	  <http://software.intel.com/en-us/mic-developer>.
+
+comment "Intel MIC Card Driver"
+
+config INTEL_MIC_CARD
+	tristate "Intel MIC Card Driver"
+	depends on 64BIT
+	default N
+	help
+	  This enables card driver support for the Intel Many Integrated
+	  Core (MIC) device family. The card driver communicates shutdown/
+	  crash events to the host and allows registration/configuration of
+	  virtio devices. Intel MIC X100 devices are currently supported.
+
+	  If you are building a card kernel for an Intel MIC device then
+	  say M (recommended) or Y, else say N. If unsure say N.
+
+	  For more information see
+	  <http://software.intel.com/en-us/mic-developer>.

drivers/misc/mic/Makefile +1 -0

index 8e72421..05b34d6 100644
@@ -3,3 +3,4 @@
 # Copyright(c) 2013, Intel Corporation.
 #
 obj-$(CONFIG_INTEL_MIC_HOST) += host/
+obj-$(CONFIG_INTEL_MIC_CARD) += card/

drivers/misc/mic/card/Makefile new file +10 -0

new file mode 100644
index 0000000..6e9675e
@@ -0,0 +1,10 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+ccflags-y += -DINTEL_MIC_CARD
+
+obj-$(CONFIG_INTEL_MIC_CARD) += mic_card.o
+mic_card-y += mic_x100.o
+mic_card-y += mic_device.o
+mic_card-y += mic_debugfs.o

drivers/misc/mic/card/mic_debugfs.c new file +130 -0

130 lines not on this page · raw message

drivers/misc/mic/card/mic_device.c new file +299 -0

299 lines not on this page · raw message

drivers/misc/mic/card/mic_device.h new file +133 -0

133 lines not on this page · raw message

drivers/misc/mic/card/mic_x100.c new file +256 -0

256 lines not on this page · raw message

drivers/misc/mic/card/mic_x100.h new file +48 -0

48 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:50 UTC #13 lore raw

PATCH v3 5/7 Intel MIC Host Driver Changes for Virtio Devices.

In reply to Sudeep Dutt

From: Ashutosh Dixit <[email protected]>

This patch introduces the host "Virtio over PCIe" interface for Intel MIC. It allows creating user space backends on the host and instantiating virtio devices for them on the Intel MIC card. It uses the existing VRINGH infrastructure in the kernel to access virtio rings from the host. A character device per MIC is exposed with IOCTL, mmap and poll callbacks. This allows the user space backend to:
(a) add/remove a virtio device via a device page.
(b) map (R/O) virtio rings and device page to user space.
(c) poll for availability of data.
(d) copy a descriptor or entire descriptor chain to/from the card.
(e) modify virtio configuration.
(f) handle virtio device reset.
The buffers are copied over using CPU copies for this initial patch and host initiated MIC DMA support is planned for future patches. The avail and desc virtio rings are in host memory and the used ring is in card memory to maximize writes across PCIe for performance.

Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 drivers/misc/mic/Kconfig             |   1 +
 drivers/misc/mic/common/mic_device.h |   7 +
 drivers/misc/mic/host/Makefile       |   2 +
 drivers/misc/mic/host/mic_boot.c     |   3 +-
 drivers/misc/mic/host/mic_debugfs.c  | 140 +++++++
 drivers/misc/mic/host/mic_device.h   |   5 +
 drivers/misc/mic/host/mic_fops.c     | 221 +++++++++++
 drivers/misc/mic/host/mic_fops.h     |  32 ++
 drivers/misc/mic/host/mic_main.c     |  27 +-
 drivers/misc/mic/host/mic_virtio.c   | 704 +++++++++++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_virtio.h   | 138 +++++++
 include/uapi/linux/Kbuild            |   1 +
 include/uapi/linux/mic_common.h      | 166 ++++++++-
 include/uapi/linux/mic_ioctl.h       |  74 ++++
 14 files changed, 1518 insertions(+), 3 deletions(-)
 create mode 100644 drivers/misc/mic/host/mic_fops.c
 create mode 100644 drivers/misc/mic/host/mic_fops.h
 create mode 100644 drivers/misc/mic/host/mic_virtio.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.h
 create mode 100644 include/uapi/linux/mic_ioctl.h

drivers/misc/mic/Kconfig +1 -0

index 279a2e6..01f1a4a 100644
@@ -3,6 +3,7 @@ comment "Intel MIC Host Driver"
 config INTEL_MIC_HOST
 	tristate "Intel MIC Host Driver"
 	depends on 64BIT && PCI
+	select VHOST_RING
 	default N
 	help
 	  This enables Host Driver support for the Intel Many Integrated

drivers/misc/mic/common/mic_device.h +7 -0

index 6440e9d..01eb74f 100644
@@ -41,4 +41,11 @@ struct mic_mw {
 #define MIC_DPLO_SPAD 14
 #define MIC_DPHI_SPAD 15
 
+/*
+ * These values are supposed to be in the config_change field of the
+ * device page when the host sends a config change interrupt to the card.
+ */
+#define MIC_VIRTIO_PARAM_DEV_REMOVE 0x1
+#define MIC_VIRTIO_PARAM_CONFIG_CHANGED 0x2
+
 #endif

drivers/misc/mic/host/Makefile +2 -0

index 98bf565..e73ac74 100644
@@ -9,3 +9,5 @@ mic_host-objs += mic_sysfs.o
 mic_host-objs += mic_smpt.o
 mic_host-objs += mic_boot.o
 mic_host-objs += mic_debugfs.o
+mic_host-objs += mic_fops.o
+mic_host-objs += mic_virtio.o

drivers/misc/mic/host/mic_boot.c +2 -1

index 936fc58..fd9ff6d 100644
@@ -20,12 +20,12 @@
  */
 #include <linux/delay.h>
 #include <linux/firmware.h>
-#include <linux/interrupt.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_smpt.h"
+#include "mic_virtio.h"
 
 /**
  * mic_reset - Reset the MIC device.

@@ -117,6 +117,7 @@ void mic_stop(struct mic_device *mdev, bool force)
 {
 	mutex_lock(&mdev->mic_mutex);
 	if (MIC_OFFLINE != mdev->state || force) {
+		mic_virtio_reset_devices(mdev);
 		mic_bootparam_init(mdev);
 		mic_reset(mdev);
 		if (MIC_RESET_FAILED == mdev->state)

drivers/misc/mic/host/mic_debugfs.c +140 -0

171 lines not on this page · raw message

drivers/misc/mic/host/mic_device.h +5 -0

index c2ca811..4592e7d 100644
@@ -21,6 +21,7 @@
 #ifndef _MIC_DEVICE_H_
 #define _MIC_DEVICE_H_
 
+#include <linux/cdev.h>
 #include <linux/idr.h>
 
 /* The maximum number of MIC devices supported in a single host system. */

@@ -156,6 +157,8 @@ struct mic_irq;
  * @dp_dma_addr: virtio device page DMA address.
  * @shutdown_db: shutdown doorbell.
  * @shutdown_cookie: shutdown cookie.
+ * @cdev: Character device for MIC.
+ * @vdev_list: list of virtio devices.
  */
 struct mic_device {
 	struct mic_mw mmio;

@@ -189,6 +192,8 @@ struct mic_device {
 	dma_addr_t dp_dma_addr;
 	int shutdown_db;
 	struct mic_irq *shutdown_cookie;
+	struct cdev cdev;
+	struct list_head vdev_list;
 };
 
 /**

drivers/misc/mic/host/mic_fops.c new file +221 -0

221 lines not on this page · raw message

drivers/misc/mic/host/mic_fops.h new file +32 -0

new file mode 100644
index 0000000..dc3893d
@@ -0,0 +1,32 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Host driver.
+ *
+ */
+#ifndef _MIC_FOPS_H_
+#define _MIC_FOPS_H_
+
+int mic_open(struct inode *inode, struct file *filp);
+int mic_release(struct inode *inode, struct file *filp);
+ssize_t mic_read(struct file *filp, char __user *buf,
+			size_t count, loff_t *pos);
+long mic_ioctl(struct file *filp, unsigned int cmd, unsigned long arg);
+int mic_mmap(struct file *f, struct vm_area_struct *vma);
+unsigned int mic_poll(struct file *f, poll_table *wait);
+
+#endif

drivers/misc/mic/host/mic_main.c +26 -1

index 2f34a26..7c4f973 100644
@@ -22,16 +22,18 @@
  * 1) Enable DMA support.
  * 2) Enable per vring interrupt support.
  */
-#include <linux/fs.h>
 #include <linux/interrupt.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/poll.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_x100.h"
 #include "mic_smpt.h"
+#include "mic_fops.h"
+#include "mic_virtio.h"
 
 static const char mic_driver_name[] = "mic";

@@ -65,6 +67,15 @@ static struct class *g_mic_class;
 /* Base device node number for MIC devices */
 static dev_t g_mic_devno;
 
+static const struct file_operations mic_fops = {
+	.open = mic_open,
+	.release = mic_release,
+	.unlocked_ioctl = mic_ioctl,
+	.poll = mic_poll,
+	.mmap = mic_mmap,
+	.owner = THIS_MODULE,
+};
+
 /* Initialize the device page */
 static int mic_dp_init(struct mic_device *mdev)
 {

@@ -814,6 +825,7 @@ mic_device_init(struct mic_device *mdev, struct pci_dev *pdev)
 	mdev->irq_info.next_avail_src = 0;
 	INIT_WORK(&mdev->reset_trigger_work, mic_reset_trigger_work);
 	INIT_WORK(&mdev->shutdown_work, mic_shutdown_work);
+	INIT_LIST_HEAD(&mdev->vdev_list);
 }
 
 /**

@@ -951,7 +963,19 @@ static int mic_probe(struct pci_dev *pdev,
 	mic_bootparam_init(mdev);
 
 	mic_create_debug_dir(mdev);
+	cdev_init(&mdev->cdev, &mic_fops);
+	mdev->cdev.owner = THIS_MODULE;
+	rc = cdev_add(&mdev->cdev, MKDEV(MAJOR(g_mic_devno), mdev->id), 1);
+	if (rc) {
+		dev_err(&pdev->dev, "cdev_add err id %d rc %d\n", mdev->id, rc);
+		goto cleanup_debug_dir;
+	}
 	return 0;
+cleanup_debug_dir:
+	mic_delete_debug_dir(mdev);
+	mutex_lock(&mdev->mic_mutex);
+	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);
+	mutex_unlock(&mdev->mic_mutex);
 dp_uninit:
 	mic_dp_uninit(mdev);
 sysfs_put:

@@ -996,6 +1020,7 @@ static void mic_remove(struct pci_dev *pdev)
 		return;
 
 	mic_stop(mdev, false);
+	cdev_del(&mdev->cdev);
 	mic_delete_debug_dir(mdev);
 	mutex_lock(&mdev->mic_mutex);
 	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);

drivers/misc/mic/host/mic_virtio.c new file +704 -0

704 lines not on this page · raw message

drivers/misc/mic/host/mic_virtio.h new file +138 -0

138 lines not on this page · raw message

include/uapi/linux/Kbuild +1 -0

index e3f7b37..c73dbd0 100644
@@ -241,6 +241,7 @@ header-y += mei.h
 header-y += mempolicy.h
 header-y += meye.h
 header-y += mic_common.h
+header-y += mic_ioctl.h
 header-y += mii.h
 header-y += minix_fs.h
 header-y += mman.h

include/uapi/linux/mic_common.h +165 -1

178 lines not on this page · raw message

include/uapi/linux/mic_ioctl.h new file +74 -0

74 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:50 UTC #22 lore raw

PATCH v3 6/7 Intel MIC Card Driver Changes for Virtio Devices.

In reply to Sudeep Dutt

From: Ashutosh Dixit <[email protected]>

This patch introduces the card "Virtio over PCIe" interface for Intel MIC. It allows virtio drivers on the card to communicate with their user space backends on the host via a device page. Ring 3 apps on the host can add, remove and configure virtio devices. A thin MIC specific virtio_config_ops is implemented which is borrowed heavily from previous similar implementations in lguest and s390 @ drivers/lguest/lguest_device.c
drivers/s390/kvm/kvm_virtio.c

Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 drivers/misc/mic/Kconfig           |   1 +
 drivers/misc/mic/card/Makefile     |   1 +
 drivers/misc/mic/card/mic_device.c |   7 +
 drivers/misc/mic/card/mic_virtio.c | 631 +++++++++++++++++++++++++++++++++++++
 drivers/misc/mic/card/mic_virtio.h |  77 +++++
 5 files changed, 717 insertions(+)
 create mode 100644 drivers/misc/mic/card/mic_virtio.c
 create mode 100644 drivers/misc/mic/card/mic_virtio.h

drivers/misc/mic/Kconfig +1 -0

index 01f1a4a..d453768 100644
@@ -24,6 +24,7 @@ comment "Intel MIC Card Driver"
 config INTEL_MIC_CARD
 	tristate "Intel MIC Card Driver"
 	depends on 64BIT
+	select VIRTIO
 	default N
 	help
 	  This enables card driver support for the Intel Many Integrated

drivers/misc/mic/card/Makefile +1 -0

index 6e9675e..69d58be 100644
@@ -8,3 +8,4 @@ obj-$(CONFIG_INTEL_MIC_CARD) += mic_card.o
 mic_card-y += mic_x100.o
 mic_card-y += mic_device.o
 mic_card-y += mic_debugfs.o
+mic_card-y += mic_virtio.o

drivers/misc/mic/card/mic_device.c +7 -0

index 3c5c302..4125217 100644
@@ -32,6 +32,7 @@
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
+#include "mic_virtio.h"
 
 static struct mic_driver *g_drv;
 static struct mic_irq *shutdown_cookie;

@@ -265,10 +266,15 @@ int __init mic_driver_init(struct mic_driver *mdrv)
 	rc = mic_shutdown_init();
 	if (rc)
 		goto irq_uninit;
+	rc = mic_devices_init(mdrv);
+	if (rc)
+		goto shutdown_uninit;
 	mic_create_card_debug_dir(mdrv);
 	atomic_notifier_chain_register(&panic_notifier_list, &mic_panic);
 done:
 	return rc;
+shutdown_uninit:
+	mic_shutdown_uninit();
 irq_uninit:
 	mic_uninit_irq();
 dp_uninit:

@@ -286,6 +292,7 @@ put:
 void mic_driver_uninit(struct mic_driver *mdrv)
 {
 	mic_delete_card_debug_dir(mdrv);
+	mic_devices_uninit(mdrv);
 	/*
 	 * Inform the host about the shutdown status i.e. poweroff/restart etc.
 	 * The module cannot be unloaded so the only code path to call

drivers/misc/mic/card/mic_virtio.c new file +631 -0

631 lines not on this page · raw message

drivers/misc/mic/card/mic_virtio.h new file +77 -0

77 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:50 UTC #23 lore raw

PATCH v3 7/7 Sample Implementation of Intel MIC User Space Daemon.

In reply to Sudeep Dutt

From: Caz Yokoyama <[email protected]>

This patch introduces a sample user space daemon which implements the virtio device backends on the host. The daemon creates/removes/configures virtio device backends by communicating with the Intel MIC Host Driver. The virtio devices currently supported are virtio net, virtio console and virtio block. Virtio net supports TSO/GSO. The daemon also monitors card shutdown status and takes appropriate actions like killing the virtio backends and resetting the card upon card shutdown and crashes.

Co-author: Ashutosh Dixit <[email protected]>
Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

---
 Documentation/mic/mic_overview.txt |   49 ++
 Documentation/mic/mpssd/.gitignore |    1 +
 Documentation/mic/mpssd/Makefile   |   19 +
 Documentation/mic/mpssd/micctrl    |  173 ++++
 Documentation/mic/mpssd/mpss       |  202 +++++
 Documentation/mic/mpssd/mpssd.c    | 1701 ++++++++++++++++++++++++++++++++++++
 Documentation/mic/mpssd/mpssd.h    |  100 +++
 Documentation/mic/mpssd/sysfs.c    |  102 +++
 8 files changed, 2347 insertions(+)
 create mode 100644 Documentation/mic/mic_overview.txt
 create mode 100644 Documentation/mic/mpssd/.gitignore
 create mode 100644 Documentation/mic/mpssd/Makefile
 create mode 100755 Documentation/mic/mpssd/micctrl
 create mode 100755 Documentation/mic/mpssd/mpss
 create mode 100644 Documentation/mic/mpssd/mpssd.c
 create mode 100644 Documentation/mic/mpssd/mpssd.h
 create mode 100644 Documentation/mic/mpssd/sysfs.c

Documentation/mic/mic_overview.txt new file +49 -0

new file mode 100644
index 0000000..c4424ed
@@ -0,0 +1,49 @@
+An Intel MIC X100 device is a PCIe form factor add-in coprocessor
+card based on the Intel Many Integrated Core (MIC) architecture
+that runs a Linux OS. It is a PCIe endpoint in a platform and therefore
+implements the three required standard address spaces i.e. configuration,
+memory and I/O. The host OS loads a device driver as is typical for
+PCIe devices. The card itself runs a bootstrap after reset that
+transfers control to the card OS downloaded from the host driver.
+The card OS as shipped by Intel is a Linux kernel with modifications
+for the X100 devices.
+
+Since it is a PCIe card, it does not have the ability to host hardware
+devices for networking, storage and console. We provide these devices
+on X100 coprocessors thus enabling a self-bootable equivalent environment
+for applications. A key benefit of our solution is that it leverages
+the standard virtio framework for network, disk and console devices,
+though in our case the virtio framework is used across a PCIe bus.
+
+Here is a block diagram of the various components described above. The
+virtio backends are situated on the host rather than the card given better
+single threaded performance for the host compared to MIC, the ability of
+the host to initiate DMA's to/from the card using the MIC DMA engine and
+the fact that the virtio block storage backend can only be on the host.
+
+                              |
+       +----------+           |             +----------+
+       | Card OS  |           |             | Host OS  |
+       +----------+           |             +----------+
+                              |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+| Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
+| Net   | |Console | |Block | | |Net      |  |Console | |Block   |
+| Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+    |         |         |     |      |            |         |
+    |         |         |     |User  |            |         |
+    |         |         |     |------|------------|---------|-------
+    +-------------------+     |Kernel +--------------------------+
+              |               |       | Virtio over PCIe IOCTLs  |
+              |               |       +--------------------------+
+      +--------------+        |                   |
+      |Intel MIC     |        |            +---------------+
+      |Card Driver   |        |            |Intel MIC      |
+      +--------------+        |            |Host Driver    |
+              |               |            +---------------+
+              |               |                   |
+     +-------------------------------------------------------------+
+     |                                                             |
+     |                    PCIe Bus                                 |
+     +-------------------------------------------------------------+

Documentation/mic/mpssd/.gitignore new file +1 -0

new file mode 100644
index 0000000..8b7c72f
@@ -0,0 +1 @@
+mpssd

Documentation/mic/mpssd/Makefile new file +19 -0

new file mode 100644
index 0000000..eb860a7
@@ -0,0 +1,19 @@
+#
+# Makefile - Intel MIC User Space Tools.
+# Copyright(c) 2013, Intel Corporation.
+#
+ifdef DEBUG
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall -DDEBUG=$(DEBUG)
+else
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall
+endif
+
+mpssd: mpssd.o sysfs.o
+	$(CC) $(CFLAGS) -o $@ $^ -lpthread
+
+install:
+	install mpssd /usr/sbin/mpssd
+	install micctrl /usr/sbin/micctrl
+
+clean:
+	rm -f mpssd *.o

Documentation/mic/mpssd/micctrl new file +173 -0

173 lines not on this page · raw message

Documentation/mic/mpssd/mpss new file +202 -0

202 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.c new file +1701 -0

1,701 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.h new file +100 -0

100 lines not on this page · raw message

Documentation/mic/mpssd/sysfs.c new file +102 -0

102 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:50 UTC #2 lore raw

PATCH v3 1/7 Intel MIC Host Driver for X100 family.

In reply to Sudeep Dutt

This patch enables the following:
a) Initializes the Intel MIC X100 PCIe devices.
b) Provides sysfs entries for family and stepping information.

Co-author: Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 Documentation/ABI/testing/sysfs-class-mic.txt |  34 +++
 drivers/misc/Kconfig                          |   1 +
 drivers/misc/Makefile                         |   1 +
 drivers/misc/mic/Kconfig                      |  19 ++
 drivers/misc/mic/Makefile                     |   5 +
 drivers/misc/mic/common/mic_device.h          |  37 +++
 drivers/misc/mic/host/Makefile                |   8 +
 drivers/misc/mic/host/mic_device.h            | 109 +++++++++
 drivers/misc/mic/host/mic_main.c              | 309 ++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_sysfs.c             |  97 ++++++++
 drivers/misc/mic/host/mic_x100.c              |  75 +++++++
 drivers/misc/mic/host/mic_x100.h              |  47 ++++
 12 files changed, 742 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt
 create mode 100644 drivers/misc/mic/Kconfig
 create mode 100644 drivers/misc/mic/Makefile
 create mode 100644 drivers/misc/mic/common/mic_device.h
 create mode 100644 drivers/misc/mic/host/Makefile
 create mode 100644 drivers/misc/mic/host/mic_device.h
 create mode 100644 drivers/misc/mic/host/mic_main.c
 create mode 100644 drivers/misc/mic/host/mic_sysfs.c
 create mode 100644 drivers/misc/mic/host/mic_x100.c
 create mode 100644 drivers/misc/mic/host/mic_x100.h

Documentation/ABI/testing/sysfs-class-mic.txt new file +34 -0

new file mode 100644
index 0000000..09eb3c6
@@ -0,0 +1,34 @@
+What:		/sys/class/mic/
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The mic class directory belongs to Intel MIC devices and
+		provides information per MIC device. An Intel MIC device is a
+		PCIe form factor add-in Coprocessor card based on the Intel Many
+		Integrated Core (MIC) architecture that runs a Linux OS.
+
+What:		/sys/class/mic/mic(x)
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The directories /sys/class/mic/mic0, /sys/class/mic/mic1 etc.,
+		represent MIC devices (0,1,..etc). Each directory has
+		information specific to that MIC device.
+
+What:		/sys/class/mic/mic(x)/family
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the Coprocessor family for an Intel
+		MIC device. For example - "x100"
+
+What:		/sys/class/mic/mic(x)/stepping
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the silicon stepping for an Intel
+		MIC device. For example - "A0" or "B0"

drivers/misc/Kconfig +1 -0

index 8dacd4c..e760715 100644
@@ -537,4 +537,5 @@ source "drivers/misc/carma/Kconfig"
 source "drivers/misc/altera-stapl/Kconfig"
 source "drivers/misc/mei/Kconfig"
 source "drivers/misc/vmw_vmci/Kconfig"
+source "drivers/misc/mic/Kconfig"
 endmenu

drivers/misc/Makefile +1 -0

index c235d5b..0b7ea3e 100644
@@ -53,3 +53,4 @@ obj-$(CONFIG_INTEL_MEI)		+= mei/
 obj-$(CONFIG_VMWARE_VMCI)	+= vmw_vmci/
 obj-$(CONFIG_LATTICE_ECP3_CONFIG)	+= lattice-ecp3-config.o
 obj-$(CONFIG_SRAM)		+= sram.o
+obj-y				+= mic/

drivers/misc/mic/Kconfig new file +19 -0

new file mode 100644
index 0000000..aaefd0c
@@ -0,0 +1,19 @@
+comment "Intel MIC Host Driver"
+
+config INTEL_MIC_HOST
+	tristate "Intel MIC Host Driver"
+	depends on 64BIT && PCI
+	default N
+	help
+	  This enables Host Driver support for the Intel Many Integrated
+	  Core (MIC) family of PCIe form factor coprocessor devices that
+	  run a 64 bit Linux OS. The driver manages card OS state and
+	  enables communication between host and card. Intel MIC X100
+	  devices are currently supported.
+
+	  If you are building a host kernel with an Intel MIC device then
+	  say M (recommended) or Y, else say N. If unsure say N.
+
+	  More information about the Intel MIC family as well as the Linux
+	  OS and tools for MIC to use with this driver are available from
+	  <http://software.intel.com/en-us/mic-developer>.

drivers/misc/mic/Makefile new file +5 -0

new file mode 100644
index 0000000..8e72421
@@ -0,0 +1,5 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += host/

drivers/misc/mic/common/mic_device.h new file +37 -0

new file mode 100644
index 0000000..f02262e
@@ -0,0 +1,37 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC driver.
+ *
+ */
+#ifndef __MIC_COMMON_DEVICE_H_
+#define __MIC_COMMON_DEVICE_H_
+
+/**
+ * struct mic_mw - MIC memory window
+ *
+ * @pa: Base physical address.
+ * @va: Base ioremap'd virtual address.
+ * @len: Size of the memory window.
+ */
+struct mic_mw {
+	phys_addr_t pa;
+	void __iomem *va;
+	resource_size_t len;
+};
+
+#endif

drivers/misc/mic/host/Makefile new file +8 -0

new file mode 100644
index 0000000..93b9d25
@@ -0,0 +1,8 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += mic_host.o
+mic_host-objs := mic_main.o
+mic_host-objs += mic_x100.o
+mic_host-objs += mic_sysfs.o

drivers/misc/mic/host/mic_device.h new file +109 -0

109 lines not on this page · raw message

drivers/misc/mic/host/mic_main.c new file +309 -0

309 lines not on this page · raw message

drivers/misc/mic/host/mic_sysfs.c new file +97 -0

97 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.c new file +75 -0

75 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.h new file +47 -0

47 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:50 UTC #14 lore raw

PATCH v3 5/7 Intel MIC Host Driver Changes for Virtio Devices.

In reply to Sudeep Dutt

From: Ashutosh Dixit <[email protected]>

Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 drivers/misc/mic/Kconfig             |   1 +
 drivers/misc/mic/common/mic_device.h |   7 +
 drivers/misc/mic/host/Makefile       |   2 +
 drivers/misc/mic/host/mic_boot.c     |   3 +-
 drivers/misc/mic/host/mic_debugfs.c  | 140 +++++++
 drivers/misc/mic/host/mic_device.h   |   5 +
 drivers/misc/mic/host/mic_fops.c     | 221 +++++++++++
 drivers/misc/mic/host/mic_fops.h     |  32 ++
 drivers/misc/mic/host/mic_main.c     |  27 +-
 drivers/misc/mic/host/mic_virtio.c   | 704 +++++++++++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_virtio.h   | 138 +++++++
 include/uapi/linux/Kbuild            |   1 +
 include/uapi/linux/mic_common.h      | 166 ++++++++-
 include/uapi/linux/mic_ioctl.h       |  74 ++++
 14 files changed, 1518 insertions(+), 3 deletions(-)
 create mode 100644 drivers/misc/mic/host/mic_fops.c
 create mode 100644 drivers/misc/mic/host/mic_fops.h
 create mode 100644 drivers/misc/mic/host/mic_virtio.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.h
 create mode 100644 include/uapi/linux/mic_ioctl.h

drivers/misc/mic/Kconfig +1 -0

index 279a2e6..01f1a4a 100644
@@ -3,6 +3,7 @@ comment "Intel MIC Host Driver"
 config INTEL_MIC_HOST
 	tristate "Intel MIC Host Driver"
 	depends on 64BIT && PCI
+	select VHOST_RING
 	default N
 	help
 	  This enables Host Driver support for the Intel Many Integrated

drivers/misc/mic/common/mic_device.h +7 -0

index 6440e9d..01eb74f 100644
@@ -41,4 +41,11 @@ struct mic_mw {
 #define MIC_DPLO_SPAD 14
 #define MIC_DPHI_SPAD 15
 
+/*
+ * These values are supposed to be in the config_change field of the
+ * device page when the host sends a config change interrupt to the card.
+ */
+#define MIC_VIRTIO_PARAM_DEV_REMOVE 0x1
+#define MIC_VIRTIO_PARAM_CONFIG_CHANGED 0x2
+
 #endif

drivers/misc/mic/host/Makefile +2 -0

index 98bf565..e73ac74 100644
@@ -9,3 +9,5 @@ mic_host-objs += mic_sysfs.o
 mic_host-objs += mic_smpt.o
 mic_host-objs += mic_boot.o
 mic_host-objs += mic_debugfs.o
+mic_host-objs += mic_fops.o
+mic_host-objs += mic_virtio.o

drivers/misc/mic/host/mic_boot.c +2 -1

index 936fc58..fd9ff6d 100644
@@ -20,12 +20,12 @@
  */
 #include <linux/delay.h>
 #include <linux/firmware.h>
-#include <linux/interrupt.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_smpt.h"
+#include "mic_virtio.h"
 
 /**
  * mic_reset - Reset the MIC device.

@@ -117,6 +117,7 @@ void mic_stop(struct mic_device *mdev, bool force)
 {
 	mutex_lock(&mdev->mic_mutex);
 	if (MIC_OFFLINE != mdev->state || force) {
+		mic_virtio_reset_devices(mdev);
 		mic_bootparam_init(mdev);
 		mic_reset(mdev);
 		if (MIC_RESET_FAILED == mdev->state)

drivers/misc/mic/host/mic_debugfs.c +140 -0

171 lines not on this page · raw message

drivers/misc/mic/host/mic_device.h +5 -0

index c2ca811..4592e7d 100644
@@ -21,6 +21,7 @@
 #ifndef _MIC_DEVICE_H_
 #define _MIC_DEVICE_H_
 
+#include <linux/cdev.h>
 #include <linux/idr.h>
 
 /* The maximum number of MIC devices supported in a single host system. */

@@ -156,6 +157,8 @@ struct mic_irq;
  * @dp_dma_addr: virtio device page DMA address.
  * @shutdown_db: shutdown doorbell.
  * @shutdown_cookie: shutdown cookie.
+ * @cdev: Character device for MIC.
+ * @vdev_list: list of virtio devices.
  */
 struct mic_device {
 	struct mic_mw mmio;

@@ -189,6 +192,8 @@ struct mic_device {
 	dma_addr_t dp_dma_addr;
 	int shutdown_db;
 	struct mic_irq *shutdown_cookie;
+	struct cdev cdev;
+	struct list_head vdev_list;
 };
 
 /**

drivers/misc/mic/host/mic_fops.c new file +221 -0

221 lines not on this page · raw message

drivers/misc/mic/host/mic_fops.h new file +32 -0

new file mode 100644
index 0000000..dc3893d
@@ -0,0 +1,32 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Host driver.
+ *
+ */
+#ifndef _MIC_FOPS_H_
+#define _MIC_FOPS_H_
+
+int mic_open(struct inode *inode, struct file *filp);
+int mic_release(struct inode *inode, struct file *filp);
+ssize_t mic_read(struct file *filp, char __user *buf,
+			size_t count, loff_t *pos);
+long mic_ioctl(struct file *filp, unsigned int cmd, unsigned long arg);
+int mic_mmap(struct file *f, struct vm_area_struct *vma);
+unsigned int mic_poll(struct file *f, poll_table *wait);
+
+#endif

drivers/misc/mic/host/mic_main.c +26 -1

index 2f34a26..7c4f973 100644
@@ -22,16 +22,18 @@
  * 1) Enable DMA support.
  * 2) Enable per vring interrupt support.
  */
-#include <linux/fs.h>
 #include <linux/interrupt.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/poll.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_x100.h"
 #include "mic_smpt.h"
+#include "mic_fops.h"
+#include "mic_virtio.h"
 
 static const char mic_driver_name[] = "mic";

@@ -65,6 +67,15 @@ static struct class *g_mic_class;
 /* Base device node number for MIC devices */
 static dev_t g_mic_devno;
 
+static const struct file_operations mic_fops = {
+	.open = mic_open,
+	.release = mic_release,
+	.unlocked_ioctl = mic_ioctl,
+	.poll = mic_poll,
+	.mmap = mic_mmap,
+	.owner = THIS_MODULE,
+};
+
 /* Initialize the device page */
 static int mic_dp_init(struct mic_device *mdev)
 {

@@ -814,6 +825,7 @@ mic_device_init(struct mic_device *mdev, struct pci_dev *pdev)
 	mdev->irq_info.next_avail_src = 0;
 	INIT_WORK(&mdev->reset_trigger_work, mic_reset_trigger_work);
 	INIT_WORK(&mdev->shutdown_work, mic_shutdown_work);
+	INIT_LIST_HEAD(&mdev->vdev_list);
 }
 
 /**

@@ -951,7 +963,19 @@ static int mic_probe(struct pci_dev *pdev,
 	mic_bootparam_init(mdev);
 
 	mic_create_debug_dir(mdev);
+	cdev_init(&mdev->cdev, &mic_fops);
+	mdev->cdev.owner = THIS_MODULE;
+	rc = cdev_add(&mdev->cdev, MKDEV(MAJOR(g_mic_devno), mdev->id), 1);
+	if (rc) {
+		dev_err(&pdev->dev, "cdev_add err id %d rc %d\n", mdev->id, rc);
+		goto cleanup_debug_dir;
+	}
 	return 0;
+cleanup_debug_dir:
+	mic_delete_debug_dir(mdev);
+	mutex_lock(&mdev->mic_mutex);
+	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);
+	mutex_unlock(&mdev->mic_mutex);
 dp_uninit:
 	mic_dp_uninit(mdev);
 sysfs_put:

@@ -996,6 +1020,7 @@ static void mic_remove(struct pci_dev *pdev)
 		return;
 
 	mic_stop(mdev, false);
+	cdev_del(&mdev->cdev);
 	mic_delete_debug_dir(mdev);
 	mutex_lock(&mdev->mic_mutex);
 	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);

drivers/misc/mic/host/mic_virtio.c new file +704 -0

704 lines not on this page · raw message

drivers/misc/mic/host/mic_virtio.h new file +138 -0

138 lines not on this page · raw message

include/uapi/linux/Kbuild +1 -0

index e3f7b37..c73dbd0 100644
@@ -241,6 +241,7 @@ header-y += mei.h
 header-y += mempolicy.h
 header-y += meye.h
 header-y += mic_common.h
+header-y += mic_ioctl.h
 header-y += mii.h
 header-y += minix_fs.h
 header-y += mman.h

include/uapi/linux/mic_common.h +165 -1

178 lines not on this page · raw message

include/uapi/linux/mic_ioctl.h new file +74 -0

74 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:50 UTC #24 lore raw

PATCH v3 7/7 Sample Implementation of Intel MIC User Space Daemon.

In reply to Sudeep Dutt

From: Caz Yokoyama <[email protected]>

Co-author: Ashutosh Dixit <[email protected]>
Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

---
 Documentation/mic/mic_overview.txt |   49 ++
 Documentation/mic/mpssd/.gitignore |    1 +
 Documentation/mic/mpssd/Makefile   |   19 +
 Documentation/mic/mpssd/micctrl    |  173 ++++
 Documentation/mic/mpssd/mpss       |  202 +++++
 Documentation/mic/mpssd/mpssd.c    | 1701 ++++++++++++++++++++++++++++++++++++
 Documentation/mic/mpssd/mpssd.h    |  100 +++
 Documentation/mic/mpssd/sysfs.c    |  102 +++
 8 files changed, 2347 insertions(+)
 create mode 100644 Documentation/mic/mic_overview.txt
 create mode 100644 Documentation/mic/mpssd/.gitignore
 create mode 100644 Documentation/mic/mpssd/Makefile
 create mode 100755 Documentation/mic/mpssd/micctrl
 create mode 100755 Documentation/mic/mpssd/mpss
 create mode 100644 Documentation/mic/mpssd/mpssd.c
 create mode 100644 Documentation/mic/mpssd/mpssd.h
 create mode 100644 Documentation/mic/mpssd/sysfs.c

Documentation/mic/mic_overview.txt new file +49 -0

new file mode 100644
index 0000000..c4424ed
@@ -0,0 +1,49 @@
+An Intel MIC X100 device is a PCIe form factor add-in coprocessor
+card based on the Intel Many Integrated Core (MIC) architecture
+that runs a Linux OS. It is a PCIe endpoint in a platform and therefore
+implements the three required standard address spaces i.e. configuration,
+memory and I/O. The host OS loads a device driver as is typical for
+PCIe devices. The card itself runs a bootstrap after reset that
+transfers control to the card OS downloaded from the host driver.
+The card OS as shipped by Intel is a Linux kernel with modifications
+for the X100 devices.
+
+Since it is a PCIe card, it does not have the ability to host hardware
+devices for networking, storage and console. We provide these devices
+on X100 coprocessors thus enabling a self-bootable equivalent environment
+for applications. A key benefit of our solution is that it leverages
+the standard virtio framework for network, disk and console devices,
+though in our case the virtio framework is used across a PCIe bus.
+
+Here is a block diagram of the various components described above. The
+virtio backends are situated on the host rather than the card given better
+single threaded performance for the host compared to MIC, the ability of
+the host to initiate DMA's to/from the card using the MIC DMA engine and
+the fact that the virtio block storage backend can only be on the host.
+
+                              |
+       +----------+           |             +----------+
+       | Card OS  |           |             | Host OS  |
+       +----------+           |             +----------+
+                              |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+| Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
+| Net   | |Console | |Block | | |Net      |  |Console | |Block   |
+| Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+    |         |         |     |      |            |         |
+    |         |         |     |User  |            |         |
+    |         |         |     |------|------------|---------|-------
+    +-------------------+     |Kernel +--------------------------+
+              |               |       | Virtio over PCIe IOCTLs  |
+              |               |       +--------------------------+
+      +--------------+        |                   |
+      |Intel MIC     |        |            +---------------+
+      |Card Driver   |        |            |Intel MIC      |
+      +--------------+        |            |Host Driver    |
+              |               |            +---------------+
+              |               |                   |
+     +-------------------------------------------------------------+
+     |                                                             |
+     |                    PCIe Bus                                 |
+     +-------------------------------------------------------------+

Documentation/mic/mpssd/.gitignore new file +1 -0

new file mode 100644
index 0000000..8b7c72f
@@ -0,0 +1 @@
+mpssd

Documentation/mic/mpssd/Makefile new file +19 -0

new file mode 100644
index 0000000..eb860a7
@@ -0,0 +1,19 @@
+#
+# Makefile - Intel MIC User Space Tools.
+# Copyright(c) 2013, Intel Corporation.
+#
+ifdef DEBUG
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall -DDEBUG=$(DEBUG)
+else
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall
+endif
+
+mpssd: mpssd.o sysfs.o
+	$(CC) $(CFLAGS) -o $@ $^ -lpthread
+
+install:
+	install mpssd /usr/sbin/mpssd
+	install micctrl /usr/sbin/micctrl
+
+clean:
+	rm -f mpssd *.o

Documentation/mic/mpssd/micctrl new file +173 -0

173 lines not on this page · raw message

Documentation/mic/mpssd/mpss new file +202 -0

202 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.c new file +1701 -0

1,701 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.h new file +100 -0

100 lines not on this page · raw message

Documentation/mic/mpssd/sysfs.c new file +102 -0

102 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:50 UTC #3 lore raw

PATCH v3 1/7 Intel MIC Host Driver for X100 family.

In reply to Sudeep Dutt

This patch enables the following:
a) Initializes the Intel MIC X100 PCIe devices.
b) Provides sysfs entries for family and stepping information.

Co-author: Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 Documentation/ABI/testing/sysfs-class-mic.txt |  34 +++
 drivers/misc/Kconfig                          |   1 +
 drivers/misc/Makefile                         |   1 +
 drivers/misc/mic/Kconfig                      |  19 ++
 drivers/misc/mic/Makefile                     |   5 +
 drivers/misc/mic/common/mic_device.h          |  37 +++
 drivers/misc/mic/host/Makefile                |   8 +
 drivers/misc/mic/host/mic_device.h            | 109 +++++++++
 drivers/misc/mic/host/mic_main.c              | 309 ++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_sysfs.c             |  97 ++++++++
 drivers/misc/mic/host/mic_x100.c              |  75 +++++++
 drivers/misc/mic/host/mic_x100.h              |  47 ++++
 12 files changed, 742 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt
 create mode 100644 drivers/misc/mic/Kconfig
 create mode 100644 drivers/misc/mic/Makefile
 create mode 100644 drivers/misc/mic/common/mic_device.h
 create mode 100644 drivers/misc/mic/host/Makefile
 create mode 100644 drivers/misc/mic/host/mic_device.h
 create mode 100644 drivers/misc/mic/host/mic_main.c
 create mode 100644 drivers/misc/mic/host/mic_sysfs.c
 create mode 100644 drivers/misc/mic/host/mic_x100.c
 create mode 100644 drivers/misc/mic/host/mic_x100.h

Documentation/ABI/testing/sysfs-class-mic.txt new file +34 -0

new file mode 100644
index 0000000..09eb3c6
@@ -0,0 +1,34 @@
+What:		/sys/class/mic/
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The mic class directory belongs to Intel MIC devices and
+		provides information per MIC device. An Intel MIC device is a
+		PCIe form factor add-in Coprocessor card based on the Intel Many
+		Integrated Core (MIC) architecture that runs a Linux OS.
+
+What:		/sys/class/mic/mic(x)
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The directories /sys/class/mic/mic0, /sys/class/mic/mic1 etc.,
+		represent MIC devices (0,1,..etc). Each directory has
+		information specific to that MIC device.
+
+What:		/sys/class/mic/mic(x)/family
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the Coprocessor family for an Intel
+		MIC device. For example - "x100"
+
+What:		/sys/class/mic/mic(x)/stepping
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the silicon stepping for an Intel
+		MIC device. For example - "A0" or "B0"

drivers/misc/Kconfig +1 -0

index 8dacd4c..e760715 100644
@@ -537,4 +537,5 @@ source "drivers/misc/carma/Kconfig"
 source "drivers/misc/altera-stapl/Kconfig"
 source "drivers/misc/mei/Kconfig"
 source "drivers/misc/vmw_vmci/Kconfig"
+source "drivers/misc/mic/Kconfig"
 endmenu

drivers/misc/Makefile +1 -0

index c235d5b..0b7ea3e 100644
@@ -53,3 +53,4 @@ obj-$(CONFIG_INTEL_MEI)		+= mei/
 obj-$(CONFIG_VMWARE_VMCI)	+= vmw_vmci/
 obj-$(CONFIG_LATTICE_ECP3_CONFIG)	+= lattice-ecp3-config.o
 obj-$(CONFIG_SRAM)		+= sram.o
+obj-y				+= mic/

drivers/misc/mic/Kconfig new file +19 -0

new file mode 100644
index 0000000..aaefd0c
@@ -0,0 +1,19 @@
+comment "Intel MIC Host Driver"
+
+config INTEL_MIC_HOST
+	tristate "Intel MIC Host Driver"
+	depends on 64BIT && PCI
+	default N
+	help
+	  This enables Host Driver support for the Intel Many Integrated
+	  Core (MIC) family of PCIe form factor coprocessor devices that
+	  run a 64 bit Linux OS. The driver manages card OS state and
+	  enables communication between host and card. Intel MIC X100
+	  devices are currently supported.
+
+	  If you are building a host kernel with an Intel MIC device then
+	  say M (recommended) or Y, else say N. If unsure say N.
+
+	  More information about the Intel MIC family as well as the Linux
+	  OS and tools for MIC to use with this driver are available from
+	  <http://software.intel.com/en-us/mic-developer>.

drivers/misc/mic/Makefile new file +5 -0

new file mode 100644
index 0000000..8e72421
@@ -0,0 +1,5 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += host/

drivers/misc/mic/common/mic_device.h new file +37 -0

new file mode 100644
index 0000000..f02262e
@@ -0,0 +1,37 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC driver.
+ *
+ */
+#ifndef __MIC_COMMON_DEVICE_H_
+#define __MIC_COMMON_DEVICE_H_
+
+/**
+ * struct mic_mw - MIC memory window
+ *
+ * @pa: Base physical address.
+ * @va: Base ioremap'd virtual address.
+ * @len: Size of the memory window.
+ */
+struct mic_mw {
+	phys_addr_t pa;
+	void __iomem *va;
+	resource_size_t len;
+};
+
+#endif

drivers/misc/mic/host/Makefile new file +8 -0

new file mode 100644
index 0000000..93b9d25
@@ -0,0 +1,8 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += mic_host.o
+mic_host-objs := mic_main.o
+mic_host-objs += mic_x100.o
+mic_host-objs += mic_sysfs.o

drivers/misc/mic/host/mic_device.h new file +109 -0

109 lines not on this page · raw message

drivers/misc/mic/host/mic_main.c new file +309 -0

309 lines not on this page · raw message

drivers/misc/mic/host/mic_sysfs.c new file +97 -0

97 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.c new file +75 -0

75 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.h new file +47 -0

47 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:51 UTC #25 lore raw

PATCH v3 7/7 Sample Implementation of Intel MIC User Space Daemon.

In reply to Sudeep Dutt

From: Caz Yokoyama <[email protected]>

Co-author: Ashutosh Dixit <[email protected]>
Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

---
 Documentation/mic/mic_overview.txt |   49 ++
 Documentation/mic/mpssd/.gitignore |    1 +
 Documentation/mic/mpssd/Makefile   |   19 +
 Documentation/mic/mpssd/micctrl    |  173 ++++
 Documentation/mic/mpssd/mpss       |  202 +++++
 Documentation/mic/mpssd/mpssd.c    | 1701 ++++++++++++++++++++++++++++++++++++
 Documentation/mic/mpssd/mpssd.h    |  100 +++
 Documentation/mic/mpssd/sysfs.c    |  102 +++
 8 files changed, 2347 insertions(+)
 create mode 100644 Documentation/mic/mic_overview.txt
 create mode 100644 Documentation/mic/mpssd/.gitignore
 create mode 100644 Documentation/mic/mpssd/Makefile
 create mode 100755 Documentation/mic/mpssd/micctrl
 create mode 100755 Documentation/mic/mpssd/mpss
 create mode 100644 Documentation/mic/mpssd/mpssd.c
 create mode 100644 Documentation/mic/mpssd/mpssd.h
 create mode 100644 Documentation/mic/mpssd/sysfs.c

Documentation/mic/mic_overview.txt new file +49 -0

new file mode 100644
index 0000000..c4424ed
@@ -0,0 +1,49 @@
+An Intel MIC X100 device is a PCIe form factor add-in coprocessor
+card based on the Intel Many Integrated Core (MIC) architecture
+that runs a Linux OS. It is a PCIe endpoint in a platform and therefore
+implements the three required standard address spaces i.e. configuration,
+memory and I/O. The host OS loads a device driver as is typical for
+PCIe devices. The card itself runs a bootstrap after reset that
+transfers control to the card OS downloaded from the host driver.
+The card OS as shipped by Intel is a Linux kernel with modifications
+for the X100 devices.
+
+Since it is a PCIe card, it does not have the ability to host hardware
+devices for networking, storage and console. We provide these devices
+on X100 coprocessors thus enabling a self-bootable equivalent environment
+for applications. A key benefit of our solution is that it leverages
+the standard virtio framework for network, disk and console devices,
+though in our case the virtio framework is used across a PCIe bus.
+
+Here is a block diagram of the various components described above. The
+virtio backends are situated on the host rather than the card given better
+single threaded performance for the host compared to MIC, the ability of
+the host to initiate DMA's to/from the card using the MIC DMA engine and
+the fact that the virtio block storage backend can only be on the host.
+
+                              |
+       +----------+           |             +----------+
+       | Card OS  |           |             | Host OS  |
+       +----------+           |             +----------+
+                              |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+| Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
+| Net   | |Console | |Block | | |Net      |  |Console | |Block   |
+| Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+    |         |         |     |      |            |         |
+    |         |         |     |User  |            |         |
+    |         |         |     |------|------------|---------|-------
+    +-------------------+     |Kernel +--------------------------+
+              |               |       | Virtio over PCIe IOCTLs  |
+              |               |       +--------------------------+
+      +--------------+        |                   |
+      |Intel MIC     |        |            +---------------+
+      |Card Driver   |        |            |Intel MIC      |
+      +--------------+        |            |Host Driver    |
+              |               |            +---------------+
+              |               |                   |
+     +-------------------------------------------------------------+
+     |                                                             |
+     |                    PCIe Bus                                 |
+     +-------------------------------------------------------------+

Documentation/mic/mpssd/.gitignore new file +1 -0

new file mode 100644
index 0000000..8b7c72f
@@ -0,0 +1 @@
+mpssd

Documentation/mic/mpssd/Makefile new file +19 -0

new file mode 100644
index 0000000..eb860a7
@@ -0,0 +1,19 @@
+#
+# Makefile - Intel MIC User Space Tools.
+# Copyright(c) 2013, Intel Corporation.
+#
+ifdef DEBUG
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall -DDEBUG=$(DEBUG)
+else
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall
+endif
+
+mpssd: mpssd.o sysfs.o
+	$(CC) $(CFLAGS) -o $@ $^ -lpthread
+
+install:
+	install mpssd /usr/sbin/mpssd
+	install micctrl /usr/sbin/micctrl
+
+clean:
+	rm -f mpssd *.o

Documentation/mic/mpssd/micctrl new file +173 -0

173 lines not on this page · raw message

Documentation/mic/mpssd/mpss new file +202 -0

202 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.c new file +1701 -0

1,701 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.h new file +100 -0

100 lines not on this page · raw message

Documentation/mic/mpssd/sysfs.c new file +102 -0

102 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:51 UTC #4 lore raw

PATCH v3 1/7 Intel MIC Host Driver for X100 family.

In reply to Sudeep Dutt

This patch enables the following:
a) Initializes the Intel MIC X100 PCIe devices.
b) Provides sysfs entries for family and stepping information.

Co-author: Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 Documentation/ABI/testing/sysfs-class-mic.txt |  34 +++
 drivers/misc/Kconfig                          |   1 +
 drivers/misc/Makefile                         |   1 +
 drivers/misc/mic/Kconfig                      |  19 ++
 drivers/misc/mic/Makefile                     |   5 +
 drivers/misc/mic/common/mic_device.h          |  37 +++
 drivers/misc/mic/host/Makefile                |   8 +
 drivers/misc/mic/host/mic_device.h            | 109 +++++++++
 drivers/misc/mic/host/mic_main.c              | 309 ++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_sysfs.c             |  97 ++++++++
 drivers/misc/mic/host/mic_x100.c              |  75 +++++++
 drivers/misc/mic/host/mic_x100.h              |  47 ++++
 12 files changed, 742 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt
 create mode 100644 drivers/misc/mic/Kconfig
 create mode 100644 drivers/misc/mic/Makefile
 create mode 100644 drivers/misc/mic/common/mic_device.h
 create mode 100644 drivers/misc/mic/host/Makefile
 create mode 100644 drivers/misc/mic/host/mic_device.h
 create mode 100644 drivers/misc/mic/host/mic_main.c
 create mode 100644 drivers/misc/mic/host/mic_sysfs.c
 create mode 100644 drivers/misc/mic/host/mic_x100.c
 create mode 100644 drivers/misc/mic/host/mic_x100.h

Documentation/ABI/testing/sysfs-class-mic.txt new file +34 -0

new file mode 100644
index 0000000..09eb3c6
@@ -0,0 +1,34 @@
+What:		/sys/class/mic/
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The mic class directory belongs to Intel MIC devices and
+		provides information per MIC device. An Intel MIC device is a
+		PCIe form factor add-in Coprocessor card based on the Intel Many
+		Integrated Core (MIC) architecture that runs a Linux OS.
+
+What:		/sys/class/mic/mic(x)
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The directories /sys/class/mic/mic0, /sys/class/mic/mic1 etc.,
+		represent MIC devices (0,1,..etc). Each directory has
+		information specific to that MIC device.
+
+What:		/sys/class/mic/mic(x)/family
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the Coprocessor family for an Intel
+		MIC device. For example - "x100"
+
+What:		/sys/class/mic/mic(x)/stepping
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the silicon stepping for an Intel
+		MIC device. For example - "A0" or "B0"

drivers/misc/Kconfig +1 -0

index 8dacd4c..e760715 100644
@@ -537,4 +537,5 @@ source "drivers/misc/carma/Kconfig"
 source "drivers/misc/altera-stapl/Kconfig"
 source "drivers/misc/mei/Kconfig"
 source "drivers/misc/vmw_vmci/Kconfig"
+source "drivers/misc/mic/Kconfig"
 endmenu

drivers/misc/Makefile +1 -0

index c235d5b..0b7ea3e 100644
@@ -53,3 +53,4 @@ obj-$(CONFIG_INTEL_MEI)		+= mei/
 obj-$(CONFIG_VMWARE_VMCI)	+= vmw_vmci/
 obj-$(CONFIG_LATTICE_ECP3_CONFIG)	+= lattice-ecp3-config.o
 obj-$(CONFIG_SRAM)		+= sram.o
+obj-y				+= mic/

drivers/misc/mic/Kconfig new file +19 -0

new file mode 100644
index 0000000..aaefd0c
@@ -0,0 +1,19 @@
+comment "Intel MIC Host Driver"
+
+config INTEL_MIC_HOST
+	tristate "Intel MIC Host Driver"
+	depends on 64BIT && PCI
+	default N
+	help
+	  This enables Host Driver support for the Intel Many Integrated
+	  Core (MIC) family of PCIe form factor coprocessor devices that
+	  run a 64 bit Linux OS. The driver manages card OS state and
+	  enables communication between host and card. Intel MIC X100
+	  devices are currently supported.
+
+	  If you are building a host kernel with an Intel MIC device then
+	  say M (recommended) or Y, else say N. If unsure say N.
+
+	  More information about the Intel MIC family as well as the Linux
+	  OS and tools for MIC to use with this driver are available from
+	  <http://software.intel.com/en-us/mic-developer>.

drivers/misc/mic/Makefile new file +5 -0

new file mode 100644
index 0000000..8e72421
@@ -0,0 +1,5 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += host/

drivers/misc/mic/common/mic_device.h new file +37 -0

new file mode 100644
index 0000000..f02262e
@@ -0,0 +1,37 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC driver.
+ *
+ */
+#ifndef __MIC_COMMON_DEVICE_H_
+#define __MIC_COMMON_DEVICE_H_
+
+/**
+ * struct mic_mw - MIC memory window
+ *
+ * @pa: Base physical address.
+ * @va: Base ioremap'd virtual address.
+ * @len: Size of the memory window.
+ */
+struct mic_mw {
+	phys_addr_t pa;
+	void __iomem *va;
+	resource_size_t len;
+};
+
+#endif

drivers/misc/mic/host/Makefile new file +8 -0

new file mode 100644
index 0000000..93b9d25
@@ -0,0 +1,8 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += mic_host.o
+mic_host-objs := mic_main.o
+mic_host-objs += mic_x100.o
+mic_host-objs += mic_sysfs.o

drivers/misc/mic/host/mic_device.h new file +109 -0

109 lines not on this page · raw message

drivers/misc/mic/host/mic_main.c new file +309 -0

309 lines not on this page · raw message

drivers/misc/mic/host/mic_sysfs.c new file +97 -0

97 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.c new file +75 -0

75 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.h new file +47 -0

47 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:51 UTC #15 lore raw

PATCH v3 5/7 Intel MIC Host Driver Changes for Virtio Devices.

In reply to Sudeep Dutt

From: Ashutosh Dixit <[email protected]>

Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 drivers/misc/mic/Kconfig             |   1 +
 drivers/misc/mic/common/mic_device.h |   7 +
 drivers/misc/mic/host/Makefile       |   2 +
 drivers/misc/mic/host/mic_boot.c     |   3 +-
 drivers/misc/mic/host/mic_debugfs.c  | 140 +++++++
 drivers/misc/mic/host/mic_device.h   |   5 +
 drivers/misc/mic/host/mic_fops.c     | 221 +++++++++++
 drivers/misc/mic/host/mic_fops.h     |  32 ++
 drivers/misc/mic/host/mic_main.c     |  27 +-
 drivers/misc/mic/host/mic_virtio.c   | 704 +++++++++++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_virtio.h   | 138 +++++++
 include/uapi/linux/Kbuild            |   1 +
 include/uapi/linux/mic_common.h      | 166 ++++++++-
 include/uapi/linux/mic_ioctl.h       |  74 ++++
 14 files changed, 1518 insertions(+), 3 deletions(-)
 create mode 100644 drivers/misc/mic/host/mic_fops.c
 create mode 100644 drivers/misc/mic/host/mic_fops.h
 create mode 100644 drivers/misc/mic/host/mic_virtio.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.h
 create mode 100644 include/uapi/linux/mic_ioctl.h

drivers/misc/mic/Kconfig +1 -0

index 279a2e6..01f1a4a 100644
@@ -3,6 +3,7 @@ comment "Intel MIC Host Driver"
 config INTEL_MIC_HOST
 	tristate "Intel MIC Host Driver"
 	depends on 64BIT && PCI
+	select VHOST_RING
 	default N
 	help
 	  This enables Host Driver support for the Intel Many Integrated

drivers/misc/mic/common/mic_device.h +7 -0

index 6440e9d..01eb74f 100644
@@ -41,4 +41,11 @@ struct mic_mw {
 #define MIC_DPLO_SPAD 14
 #define MIC_DPHI_SPAD 15
 
+/*
+ * These values are supposed to be in the config_change field of the
+ * device page when the host sends a config change interrupt to the card.
+ */
+#define MIC_VIRTIO_PARAM_DEV_REMOVE 0x1
+#define MIC_VIRTIO_PARAM_CONFIG_CHANGED 0x2
+
 #endif

drivers/misc/mic/host/Makefile +2 -0

index 98bf565..e73ac74 100644
@@ -9,3 +9,5 @@ mic_host-objs += mic_sysfs.o
 mic_host-objs += mic_smpt.o
 mic_host-objs += mic_boot.o
 mic_host-objs += mic_debugfs.o
+mic_host-objs += mic_fops.o
+mic_host-objs += mic_virtio.o

drivers/misc/mic/host/mic_boot.c +2 -1

index 936fc58..fd9ff6d 100644
@@ -20,12 +20,12 @@
  */
 #include <linux/delay.h>
 #include <linux/firmware.h>
-#include <linux/interrupt.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_smpt.h"
+#include "mic_virtio.h"
 
 /**
  * mic_reset - Reset the MIC device.

@@ -117,6 +117,7 @@ void mic_stop(struct mic_device *mdev, bool force)
 {
 	mutex_lock(&mdev->mic_mutex);
 	if (MIC_OFFLINE != mdev->state || force) {
+		mic_virtio_reset_devices(mdev);
 		mic_bootparam_init(mdev);
 		mic_reset(mdev);
 		if (MIC_RESET_FAILED == mdev->state)

drivers/misc/mic/host/mic_debugfs.c +140 -0

171 lines not on this page · raw message

drivers/misc/mic/host/mic_device.h +5 -0

index c2ca811..4592e7d 100644
@@ -21,6 +21,7 @@
 #ifndef _MIC_DEVICE_H_
 #define _MIC_DEVICE_H_
 
+#include <linux/cdev.h>
 #include <linux/idr.h>
 
 /* The maximum number of MIC devices supported in a single host system. */

@@ -156,6 +157,8 @@ struct mic_irq;
  * @dp_dma_addr: virtio device page DMA address.
  * @shutdown_db: shutdown doorbell.
  * @shutdown_cookie: shutdown cookie.
+ * @cdev: Character device for MIC.
+ * @vdev_list: list of virtio devices.
  */
 struct mic_device {
 	struct mic_mw mmio;

@@ -189,6 +192,8 @@ struct mic_device {
 	dma_addr_t dp_dma_addr;
 	int shutdown_db;
 	struct mic_irq *shutdown_cookie;
+	struct cdev cdev;
+	struct list_head vdev_list;
 };
 
 /**

drivers/misc/mic/host/mic_fops.c new file +221 -0

221 lines not on this page · raw message

drivers/misc/mic/host/mic_fops.h new file +32 -0

new file mode 100644
index 0000000..dc3893d
@@ -0,0 +1,32 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Host driver.
+ *
+ */
+#ifndef _MIC_FOPS_H_
+#define _MIC_FOPS_H_
+
+int mic_open(struct inode *inode, struct file *filp);
+int mic_release(struct inode *inode, struct file *filp);
+ssize_t mic_read(struct file *filp, char __user *buf,
+			size_t count, loff_t *pos);
+long mic_ioctl(struct file *filp, unsigned int cmd, unsigned long arg);
+int mic_mmap(struct file *f, struct vm_area_struct *vma);
+unsigned int mic_poll(struct file *f, poll_table *wait);
+
+#endif

drivers/misc/mic/host/mic_main.c +26 -1

index 2f34a26..7c4f973 100644
@@ -22,16 +22,18 @@
  * 1) Enable DMA support.
  * 2) Enable per vring interrupt support.
  */
-#include <linux/fs.h>
 #include <linux/interrupt.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/poll.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_x100.h"
 #include "mic_smpt.h"
+#include "mic_fops.h"
+#include "mic_virtio.h"
 
 static const char mic_driver_name[] = "mic";

@@ -65,6 +67,15 @@ static struct class *g_mic_class;
 /* Base device node number for MIC devices */
 static dev_t g_mic_devno;
 
+static const struct file_operations mic_fops = {
+	.open = mic_open,
+	.release = mic_release,
+	.unlocked_ioctl = mic_ioctl,
+	.poll = mic_poll,
+	.mmap = mic_mmap,
+	.owner = THIS_MODULE,
+};
+
 /* Initialize the device page */
 static int mic_dp_init(struct mic_device *mdev)
 {

@@ -814,6 +825,7 @@ mic_device_init(struct mic_device *mdev, struct pci_dev *pdev)
 	mdev->irq_info.next_avail_src = 0;
 	INIT_WORK(&mdev->reset_trigger_work, mic_reset_trigger_work);
 	INIT_WORK(&mdev->shutdown_work, mic_shutdown_work);
+	INIT_LIST_HEAD(&mdev->vdev_list);
 }
 
 /**

@@ -951,7 +963,19 @@ static int mic_probe(struct pci_dev *pdev,
 	mic_bootparam_init(mdev);
 
 	mic_create_debug_dir(mdev);
+	cdev_init(&mdev->cdev, &mic_fops);
+	mdev->cdev.owner = THIS_MODULE;
+	rc = cdev_add(&mdev->cdev, MKDEV(MAJOR(g_mic_devno), mdev->id), 1);
+	if (rc) {
+		dev_err(&pdev->dev, "cdev_add err id %d rc %d\n", mdev->id, rc);
+		goto cleanup_debug_dir;
+	}
 	return 0;
+cleanup_debug_dir:
+	mic_delete_debug_dir(mdev);
+	mutex_lock(&mdev->mic_mutex);
+	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);
+	mutex_unlock(&mdev->mic_mutex);
 dp_uninit:
 	mic_dp_uninit(mdev);
 sysfs_put:

@@ -996,6 +1020,7 @@ static void mic_remove(struct pci_dev *pdev)
 		return;
 
 	mic_stop(mdev, false);
+	cdev_del(&mdev->cdev);
 	mic_delete_debug_dir(mdev);
 	mutex_lock(&mdev->mic_mutex);
 	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);

drivers/misc/mic/host/mic_virtio.c new file +704 -0

704 lines not on this page · raw message

drivers/misc/mic/host/mic_virtio.h new file +138 -0

138 lines not on this page · raw message

include/uapi/linux/Kbuild +1 -0

index e3f7b37..c73dbd0 100644
@@ -241,6 +241,7 @@ header-y += mei.h
 header-y += mempolicy.h
 header-y += meye.h
 header-y += mic_common.h
+header-y += mic_ioctl.h
 header-y += mii.h
 header-y += minix_fs.h
 header-y += mman.h

include/uapi/linux/mic_common.h +165 -1

178 lines not on this page · raw message

include/uapi/linux/mic_ioctl.h new file +74 -0

74 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:51 UTC #26 lore raw

PATCH v3 7/7 Sample Implementation of Intel MIC User Space Daemon.

In reply to Sudeep Dutt

From: Caz Yokoyama <[email protected]>

Co-author: Ashutosh Dixit <[email protected]>
Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

---
 Documentation/mic/mic_overview.txt |   49 ++
 Documentation/mic/mpssd/.gitignore |    1 +
 Documentation/mic/mpssd/Makefile   |   19 +
 Documentation/mic/mpssd/micctrl    |  173 ++++
 Documentation/mic/mpssd/mpss       |  202 +++++
 Documentation/mic/mpssd/mpssd.c    | 1701 ++++++++++++++++++++++++++++++++++++
 Documentation/mic/mpssd/mpssd.h    |  100 +++
 Documentation/mic/mpssd/sysfs.c    |  102 +++
 8 files changed, 2347 insertions(+)
 create mode 100644 Documentation/mic/mic_overview.txt
 create mode 100644 Documentation/mic/mpssd/.gitignore
 create mode 100644 Documentation/mic/mpssd/Makefile
 create mode 100755 Documentation/mic/mpssd/micctrl
 create mode 100755 Documentation/mic/mpssd/mpss
 create mode 100644 Documentation/mic/mpssd/mpssd.c
 create mode 100644 Documentation/mic/mpssd/mpssd.h
 create mode 100644 Documentation/mic/mpssd/sysfs.c

Documentation/mic/mic_overview.txt new file +49 -0

new file mode 100644
index 0000000..c4424ed
@@ -0,0 +1,49 @@
+An Intel MIC X100 device is a PCIe form factor add-in coprocessor
+card based on the Intel Many Integrated Core (MIC) architecture
+that runs a Linux OS. It is a PCIe endpoint in a platform and therefore
+implements the three required standard address spaces i.e. configuration,
+memory and I/O. The host OS loads a device driver as is typical for
+PCIe devices. The card itself runs a bootstrap after reset that
+transfers control to the card OS downloaded from the host driver.
+The card OS as shipped by Intel is a Linux kernel with modifications
+for the X100 devices.
+
+Since it is a PCIe card, it does not have the ability to host hardware
+devices for networking, storage and console. We provide these devices
+on X100 coprocessors thus enabling a self-bootable equivalent environment
+for applications. A key benefit of our solution is that it leverages
+the standard virtio framework for network, disk and console devices,
+though in our case the virtio framework is used across a PCIe bus.
+
+Here is a block diagram of the various components described above. The
+virtio backends are situated on the host rather than the card given better
+single threaded performance for the host compared to MIC, the ability of
+the host to initiate DMA's to/from the card using the MIC DMA engine and
+the fact that the virtio block storage backend can only be on the host.
+
+                              |
+       +----------+           |             +----------+
+       | Card OS  |           |             | Host OS  |
+       +----------+           |             +----------+
+                              |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+| Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
+| Net   | |Console | |Block | | |Net      |  |Console | |Block   |
+| Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+    |         |         |     |      |            |         |
+    |         |         |     |User  |            |         |
+    |         |         |     |------|------------|---------|-------
+    +-------------------+     |Kernel +--------------------------+
+              |               |       | Virtio over PCIe IOCTLs  |
+              |               |       +--------------------------+
+      +--------------+        |                   |
+      |Intel MIC     |        |            +---------------+
+      |Card Driver   |        |            |Intel MIC      |
+      +--------------+        |            |Host Driver    |
+              |               |            +---------------+
+              |               |                   |
+     +-------------------------------------------------------------+
+     |                                                             |
+     |                    PCIe Bus                                 |
+     +-------------------------------------------------------------+

Documentation/mic/mpssd/.gitignore new file +1 -0

new file mode 100644
index 0000000..8b7c72f
@@ -0,0 +1 @@
+mpssd

Documentation/mic/mpssd/Makefile new file +19 -0

new file mode 100644
index 0000000..eb860a7
@@ -0,0 +1,19 @@
+#
+# Makefile - Intel MIC User Space Tools.
+# Copyright(c) 2013, Intel Corporation.
+#
+ifdef DEBUG
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall -DDEBUG=$(DEBUG)
+else
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall
+endif
+
+mpssd: mpssd.o sysfs.o
+	$(CC) $(CFLAGS) -o $@ $^ -lpthread
+
+install:
+	install mpssd /usr/sbin/mpssd
+	install micctrl /usr/sbin/micctrl
+
+clean:
+	rm -f mpssd *.o

Documentation/mic/mpssd/micctrl new file +173 -0

173 lines not on this page · raw message

Documentation/mic/mpssd/mpss new file +202 -0

202 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.c new file +1701 -0

1,701 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.h new file +100 -0

100 lines not on this page · raw message

Documentation/mic/mpssd/sysfs.c new file +102 -0

102 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:51 UTC #5 lore raw

PATCH v3 1/7 Intel MIC Host Driver for X100 family.

In reply to Sudeep Dutt

This patch enables the following:
a) Initializes the Intel MIC X100 PCIe devices.
b) Provides sysfs entries for family and stepping information.

Co-author: Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 Documentation/ABI/testing/sysfs-class-mic.txt |  34 +++
 drivers/misc/Kconfig                          |   1 +
 drivers/misc/Makefile                         |   1 +
 drivers/misc/mic/Kconfig                      |  19 ++
 drivers/misc/mic/Makefile                     |   5 +
 drivers/misc/mic/common/mic_device.h          |  37 +++
 drivers/misc/mic/host/Makefile                |   8 +
 drivers/misc/mic/host/mic_device.h            | 109 +++++++++
 drivers/misc/mic/host/mic_main.c              | 309 ++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_sysfs.c             |  97 ++++++++
 drivers/misc/mic/host/mic_x100.c              |  75 +++++++
 drivers/misc/mic/host/mic_x100.h              |  47 ++++
 12 files changed, 742 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt
 create mode 100644 drivers/misc/mic/Kconfig
 create mode 100644 drivers/misc/mic/Makefile
 create mode 100644 drivers/misc/mic/common/mic_device.h
 create mode 100644 drivers/misc/mic/host/Makefile
 create mode 100644 drivers/misc/mic/host/mic_device.h
 create mode 100644 drivers/misc/mic/host/mic_main.c
 create mode 100644 drivers/misc/mic/host/mic_sysfs.c
 create mode 100644 drivers/misc/mic/host/mic_x100.c
 create mode 100644 drivers/misc/mic/host/mic_x100.h

Documentation/ABI/testing/sysfs-class-mic.txt new file +34 -0

new file mode 100644
index 0000000..09eb3c6
@@ -0,0 +1,34 @@
+What:		/sys/class/mic/
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The mic class directory belongs to Intel MIC devices and
+		provides information per MIC device. An Intel MIC device is a
+		PCIe form factor add-in Coprocessor card based on the Intel Many
+		Integrated Core (MIC) architecture that runs a Linux OS.
+
+What:		/sys/class/mic/mic(x)
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The directories /sys/class/mic/mic0, /sys/class/mic/mic1 etc.,
+		represent MIC devices (0,1,..etc). Each directory has
+		information specific to that MIC device.
+
+What:		/sys/class/mic/mic(x)/family
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the Coprocessor family for an Intel
+		MIC device. For example - "x100"
+
+What:		/sys/class/mic/mic(x)/stepping
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the silicon stepping for an Intel
+		MIC device. For example - "A0" or "B0"

drivers/misc/Kconfig +1 -0

index 8dacd4c..e760715 100644
@@ -537,4 +537,5 @@ source "drivers/misc/carma/Kconfig"
 source "drivers/misc/altera-stapl/Kconfig"
 source "drivers/misc/mei/Kconfig"
 source "drivers/misc/vmw_vmci/Kconfig"
+source "drivers/misc/mic/Kconfig"
 endmenu

drivers/misc/Makefile +1 -0

index c235d5b..0b7ea3e 100644
@@ -53,3 +53,4 @@ obj-$(CONFIG_INTEL_MEI)		+= mei/
 obj-$(CONFIG_VMWARE_VMCI)	+= vmw_vmci/
 obj-$(CONFIG_LATTICE_ECP3_CONFIG)	+= lattice-ecp3-config.o
 obj-$(CONFIG_SRAM)		+= sram.o
+obj-y				+= mic/

drivers/misc/mic/Kconfig new file +19 -0

new file mode 100644
index 0000000..aaefd0c
@@ -0,0 +1,19 @@
+comment "Intel MIC Host Driver"
+
+config INTEL_MIC_HOST
+	tristate "Intel MIC Host Driver"
+	depends on 64BIT && PCI
+	default N
+	help
+	  This enables Host Driver support for the Intel Many Integrated
+	  Core (MIC) family of PCIe form factor coprocessor devices that
+	  run a 64 bit Linux OS. The driver manages card OS state and
+	  enables communication between host and card. Intel MIC X100
+	  devices are currently supported.
+
+	  If you are building a host kernel with an Intel MIC device then
+	  say M (recommended) or Y, else say N. If unsure say N.
+
+	  More information about the Intel MIC family as well as the Linux
+	  OS and tools for MIC to use with this driver are available from
+	  <http://software.intel.com/en-us/mic-developer>.

drivers/misc/mic/Makefile new file +5 -0

new file mode 100644
index 0000000..8e72421
@@ -0,0 +1,5 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += host/

drivers/misc/mic/common/mic_device.h new file +37 -0

new file mode 100644
index 0000000..f02262e
@@ -0,0 +1,37 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC driver.
+ *
+ */
+#ifndef __MIC_COMMON_DEVICE_H_
+#define __MIC_COMMON_DEVICE_H_
+
+/**
+ * struct mic_mw - MIC memory window
+ *
+ * @pa: Base physical address.
+ * @va: Base ioremap'd virtual address.
+ * @len: Size of the memory window.
+ */
+struct mic_mw {
+	phys_addr_t pa;
+	void __iomem *va;
+	resource_size_t len;
+};
+
+#endif

drivers/misc/mic/host/Makefile new file +8 -0

new file mode 100644
index 0000000..93b9d25
@@ -0,0 +1,8 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += mic_host.o
+mic_host-objs := mic_main.o
+mic_host-objs += mic_x100.o
+mic_host-objs += mic_sysfs.o

drivers/misc/mic/host/mic_device.h new file +109 -0

109 lines not on this page · raw message

drivers/misc/mic/host/mic_main.c new file +309 -0

309 lines not on this page · raw message

drivers/misc/mic/host/mic_sysfs.c new file +97 -0

97 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.c new file +75 -0

75 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.h new file +47 -0

47 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:51 UTC #16 lore raw

PATCH v3 5/7 Intel MIC Host Driver Changes for Virtio Devices.

In reply to Sudeep Dutt

From: Ashutosh Dixit <[email protected]>

Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 drivers/misc/mic/Kconfig             |   1 +
 drivers/misc/mic/common/mic_device.h |   7 +
 drivers/misc/mic/host/Makefile       |   2 +
 drivers/misc/mic/host/mic_boot.c     |   3 +-
 drivers/misc/mic/host/mic_debugfs.c  | 140 +++++++
 drivers/misc/mic/host/mic_device.h   |   5 +
 drivers/misc/mic/host/mic_fops.c     | 221 +++++++++++
 drivers/misc/mic/host/mic_fops.h     |  32 ++
 drivers/misc/mic/host/mic_main.c     |  27 +-
 drivers/misc/mic/host/mic_virtio.c   | 704 +++++++++++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_virtio.h   | 138 +++++++
 include/uapi/linux/Kbuild            |   1 +
 include/uapi/linux/mic_common.h      | 166 ++++++++-
 include/uapi/linux/mic_ioctl.h       |  74 ++++
 14 files changed, 1518 insertions(+), 3 deletions(-)
 create mode 100644 drivers/misc/mic/host/mic_fops.c
 create mode 100644 drivers/misc/mic/host/mic_fops.h
 create mode 100644 drivers/misc/mic/host/mic_virtio.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.h
 create mode 100644 include/uapi/linux/mic_ioctl.h

drivers/misc/mic/Kconfig +1 -0

index 279a2e6..01f1a4a 100644
@@ -3,6 +3,7 @@ comment "Intel MIC Host Driver"
 config INTEL_MIC_HOST
 	tristate "Intel MIC Host Driver"
 	depends on 64BIT && PCI
+	select VHOST_RING
 	default N
 	help
 	  This enables Host Driver support for the Intel Many Integrated

drivers/misc/mic/common/mic_device.h +7 -0

index 6440e9d..01eb74f 100644
@@ -41,4 +41,11 @@ struct mic_mw {
 #define MIC_DPLO_SPAD 14
 #define MIC_DPHI_SPAD 15
 
+/*
+ * These values are supposed to be in the config_change field of the
+ * device page when the host sends a config change interrupt to the card.
+ */
+#define MIC_VIRTIO_PARAM_DEV_REMOVE 0x1
+#define MIC_VIRTIO_PARAM_CONFIG_CHANGED 0x2
+
 #endif

drivers/misc/mic/host/Makefile +2 -0

index 98bf565..e73ac74 100644
@@ -9,3 +9,5 @@ mic_host-objs += mic_sysfs.o
 mic_host-objs += mic_smpt.o
 mic_host-objs += mic_boot.o
 mic_host-objs += mic_debugfs.o
+mic_host-objs += mic_fops.o
+mic_host-objs += mic_virtio.o

drivers/misc/mic/host/mic_boot.c +2 -1

index 936fc58..fd9ff6d 100644
@@ -20,12 +20,12 @@
  */
 #include <linux/delay.h>
 #include <linux/firmware.h>
-#include <linux/interrupt.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_smpt.h"
+#include "mic_virtio.h"
 
 /**
  * mic_reset - Reset the MIC device.

@@ -117,6 +117,7 @@ void mic_stop(struct mic_device *mdev, bool force)
 {
 	mutex_lock(&mdev->mic_mutex);
 	if (MIC_OFFLINE != mdev->state || force) {
+		mic_virtio_reset_devices(mdev);
 		mic_bootparam_init(mdev);
 		mic_reset(mdev);
 		if (MIC_RESET_FAILED == mdev->state)

drivers/misc/mic/host/mic_debugfs.c +140 -0

171 lines not on this page · raw message

drivers/misc/mic/host/mic_device.h +5 -0

index c2ca811..4592e7d 100644
@@ -21,6 +21,7 @@
 #ifndef _MIC_DEVICE_H_
 #define _MIC_DEVICE_H_
 
+#include <linux/cdev.h>
 #include <linux/idr.h>
 
 /* The maximum number of MIC devices supported in a single host system. */

@@ -156,6 +157,8 @@ struct mic_irq;
  * @dp_dma_addr: virtio device page DMA address.
  * @shutdown_db: shutdown doorbell.
  * @shutdown_cookie: shutdown cookie.
+ * @cdev: Character device for MIC.
+ * @vdev_list: list of virtio devices.
  */
 struct mic_device {
 	struct mic_mw mmio;

@@ -189,6 +192,8 @@ struct mic_device {
 	dma_addr_t dp_dma_addr;
 	int shutdown_db;
 	struct mic_irq *shutdown_cookie;
+	struct cdev cdev;
+	struct list_head vdev_list;
 };
 
 /**

drivers/misc/mic/host/mic_fops.c new file +221 -0

221 lines not on this page · raw message

drivers/misc/mic/host/mic_fops.h new file +32 -0

new file mode 100644
index 0000000..dc3893d
@@ -0,0 +1,32 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Host driver.
+ *
+ */
+#ifndef _MIC_FOPS_H_
+#define _MIC_FOPS_H_
+
+int mic_open(struct inode *inode, struct file *filp);
+int mic_release(struct inode *inode, struct file *filp);
+ssize_t mic_read(struct file *filp, char __user *buf,
+			size_t count, loff_t *pos);
+long mic_ioctl(struct file *filp, unsigned int cmd, unsigned long arg);
+int mic_mmap(struct file *f, struct vm_area_struct *vma);
+unsigned int mic_poll(struct file *f, poll_table *wait);
+
+#endif

drivers/misc/mic/host/mic_main.c +26 -1

index 2f34a26..7c4f973 100644
@@ -22,16 +22,18 @@
  * 1) Enable DMA support.
  * 2) Enable per vring interrupt support.
  */
-#include <linux/fs.h>
 #include <linux/interrupt.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/poll.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_x100.h"
 #include "mic_smpt.h"
+#include "mic_fops.h"
+#include "mic_virtio.h"
 
 static const char mic_driver_name[] = "mic";

@@ -65,6 +67,15 @@ static struct class *g_mic_class;
 /* Base device node number for MIC devices */
 static dev_t g_mic_devno;
 
+static const struct file_operations mic_fops = {
+	.open = mic_open,
+	.release = mic_release,
+	.unlocked_ioctl = mic_ioctl,
+	.poll = mic_poll,
+	.mmap = mic_mmap,
+	.owner = THIS_MODULE,
+};
+
 /* Initialize the device page */
 static int mic_dp_init(struct mic_device *mdev)
 {

@@ -814,6 +825,7 @@ mic_device_init(struct mic_device *mdev, struct pci_dev *pdev)
 	mdev->irq_info.next_avail_src = 0;
 	INIT_WORK(&mdev->reset_trigger_work, mic_reset_trigger_work);
 	INIT_WORK(&mdev->shutdown_work, mic_shutdown_work);
+	INIT_LIST_HEAD(&mdev->vdev_list);
 }
 
 /**

@@ -951,7 +963,19 @@ static int mic_probe(struct pci_dev *pdev,
 	mic_bootparam_init(mdev);
 
 	mic_create_debug_dir(mdev);
+	cdev_init(&mdev->cdev, &mic_fops);
+	mdev->cdev.owner = THIS_MODULE;
+	rc = cdev_add(&mdev->cdev, MKDEV(MAJOR(g_mic_devno), mdev->id), 1);
+	if (rc) {
+		dev_err(&pdev->dev, "cdev_add err id %d rc %d\n", mdev->id, rc);
+		goto cleanup_debug_dir;
+	}
 	return 0;
+cleanup_debug_dir:
+	mic_delete_debug_dir(mdev);
+	mutex_lock(&mdev->mic_mutex);
+	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);
+	mutex_unlock(&mdev->mic_mutex);
 dp_uninit:
 	mic_dp_uninit(mdev);
 sysfs_put:

@@ -996,6 +1020,7 @@ static void mic_remove(struct pci_dev *pdev)
 		return;
 
 	mic_stop(mdev, false);
+	cdev_del(&mdev->cdev);
 	mic_delete_debug_dir(mdev);
 	mutex_lock(&mdev->mic_mutex);
 	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);

drivers/misc/mic/host/mic_virtio.c new file +704 -0

704 lines not on this page · raw message

drivers/misc/mic/host/mic_virtio.h new file +138 -0

138 lines not on this page · raw message

include/uapi/linux/Kbuild +1 -0

index e3f7b37..c73dbd0 100644
@@ -241,6 +241,7 @@ header-y += mei.h
 header-y += mempolicy.h
 header-y += meye.h
 header-y += mic_common.h
+header-y += mic_ioctl.h
 header-y += mii.h
 header-y += minix_fs.h
 header-y += mman.h

include/uapi/linux/mic_common.h +165 -1

178 lines not on this page · raw message

include/uapi/linux/mic_ioctl.h new file +74 -0

74 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:51 UTC #27 lore raw

PATCH v3 7/7 Sample Implementation of Intel MIC User Space Daemon.

In reply to Sudeep Dutt

From: Caz Yokoyama <[email protected]>

Co-author: Ashutosh Dixit <[email protected]>
Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

---
 Documentation/mic/mic_overview.txt |   49 ++
 Documentation/mic/mpssd/.gitignore |    1 +
 Documentation/mic/mpssd/Makefile   |   19 +
 Documentation/mic/mpssd/micctrl    |  173 ++++
 Documentation/mic/mpssd/mpss       |  202 +++++
 Documentation/mic/mpssd/mpssd.c    | 1701 ++++++++++++++++++++++++++++++++++++
 Documentation/mic/mpssd/mpssd.h    |  100 +++
 Documentation/mic/mpssd/sysfs.c    |  102 +++
 8 files changed, 2347 insertions(+)
 create mode 100644 Documentation/mic/mic_overview.txt
 create mode 100644 Documentation/mic/mpssd/.gitignore
 create mode 100644 Documentation/mic/mpssd/Makefile
 create mode 100755 Documentation/mic/mpssd/micctrl
 create mode 100755 Documentation/mic/mpssd/mpss
 create mode 100644 Documentation/mic/mpssd/mpssd.c
 create mode 100644 Documentation/mic/mpssd/mpssd.h
 create mode 100644 Documentation/mic/mpssd/sysfs.c

Documentation/mic/mic_overview.txt new file +49 -0

new file mode 100644
index 0000000..c4424ed
@@ -0,0 +1,49 @@
+An Intel MIC X100 device is a PCIe form factor add-in coprocessor
+card based on the Intel Many Integrated Core (MIC) architecture
+that runs a Linux OS. It is a PCIe endpoint in a platform and therefore
+implements the three required standard address spaces i.e. configuration,
+memory and I/O. The host OS loads a device driver as is typical for
+PCIe devices. The card itself runs a bootstrap after reset that
+transfers control to the card OS downloaded from the host driver.
+The card OS as shipped by Intel is a Linux kernel with modifications
+for the X100 devices.
+
+Since it is a PCIe card, it does not have the ability to host hardware
+devices for networking, storage and console. We provide these devices
+on X100 coprocessors thus enabling a self-bootable equivalent environment
+for applications. A key benefit of our solution is that it leverages
+the standard virtio framework for network, disk and console devices,
+though in our case the virtio framework is used across a PCIe bus.
+
+Here is a block diagram of the various components described above. The
+virtio backends are situated on the host rather than the card given better
+single threaded performance for the host compared to MIC, the ability of
+the host to initiate DMA's to/from the card using the MIC DMA engine and
+the fact that the virtio block storage backend can only be on the host.
+
+                              |
+       +----------+           |             +----------+
+       | Card OS  |           |             | Host OS  |
+       +----------+           |             +----------+
+                              |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+| Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
+| Net   | |Console | |Block | | |Net      |  |Console | |Block   |
+| Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+    |         |         |     |      |            |         |
+    |         |         |     |User  |            |         |
+    |         |         |     |------|------------|---------|-------
+    +-------------------+     |Kernel +--------------------------+
+              |               |       | Virtio over PCIe IOCTLs  |
+              |               |       +--------------------------+
+      +--------------+        |                   |
+      |Intel MIC     |        |            +---------------+
+      |Card Driver   |        |            |Intel MIC      |
+      +--------------+        |            |Host Driver    |
+              |               |            +---------------+
+              |               |                   |
+     +-------------------------------------------------------------+
+     |                                                             |
+     |                    PCIe Bus                                 |
+     +-------------------------------------------------------------+

Documentation/mic/mpssd/.gitignore new file +1 -0

new file mode 100644
index 0000000..8b7c72f
@@ -0,0 +1 @@
+mpssd

Documentation/mic/mpssd/Makefile new file +19 -0

new file mode 100644
index 0000000..eb860a7
@@ -0,0 +1,19 @@
+#
+# Makefile - Intel MIC User Space Tools.
+# Copyright(c) 2013, Intel Corporation.
+#
+ifdef DEBUG
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall -DDEBUG=$(DEBUG)
+else
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall
+endif
+
+mpssd: mpssd.o sysfs.o
+	$(CC) $(CFLAGS) -o $@ $^ -lpthread
+
+install:
+	install mpssd /usr/sbin/mpssd
+	install micctrl /usr/sbin/micctrl
+
+clean:
+	rm -f mpssd *.o

Documentation/mic/mpssd/micctrl new file +173 -0

173 lines not on this page · raw message

Documentation/mic/mpssd/mpss new file +202 -0

202 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.c new file +1701 -0

1,701 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.h new file +100 -0

100 lines not on this page · raw message

Documentation/mic/mpssd/sysfs.c new file +102 -0

102 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:51 UTC #6 lore raw

PATCH v3 1/7 Intel MIC Host Driver for X100 family.

In reply to Sudeep Dutt

This patch enables the following:
a) Initializes the Intel MIC X100 PCIe devices.
b) Provides sysfs entries for family and stepping information.

Co-author: Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 Documentation/ABI/testing/sysfs-class-mic.txt |  34 +++
 drivers/misc/Kconfig                          |   1 +
 drivers/misc/Makefile                         |   1 +
 drivers/misc/mic/Kconfig                      |  19 ++
 drivers/misc/mic/Makefile                     |   5 +
 drivers/misc/mic/common/mic_device.h          |  37 +++
 drivers/misc/mic/host/Makefile                |   8 +
 drivers/misc/mic/host/mic_device.h            | 109 +++++++++
 drivers/misc/mic/host/mic_main.c              | 309 ++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_sysfs.c             |  97 ++++++++
 drivers/misc/mic/host/mic_x100.c              |  75 +++++++
 drivers/misc/mic/host/mic_x100.h              |  47 ++++
 12 files changed, 742 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt
 create mode 100644 drivers/misc/mic/Kconfig
 create mode 100644 drivers/misc/mic/Makefile
 create mode 100644 drivers/misc/mic/common/mic_device.h
 create mode 100644 drivers/misc/mic/host/Makefile
 create mode 100644 drivers/misc/mic/host/mic_device.h
 create mode 100644 drivers/misc/mic/host/mic_main.c
 create mode 100644 drivers/misc/mic/host/mic_sysfs.c
 create mode 100644 drivers/misc/mic/host/mic_x100.c
 create mode 100644 drivers/misc/mic/host/mic_x100.h

Documentation/ABI/testing/sysfs-class-mic.txt new file +34 -0

new file mode 100644
index 0000000..09eb3c6
@@ -0,0 +1,34 @@
+What:		/sys/class/mic/
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The mic class directory belongs to Intel MIC devices and
+		provides information per MIC device. An Intel MIC device is a
+		PCIe form factor add-in Coprocessor card based on the Intel Many
+		Integrated Core (MIC) architecture that runs a Linux OS.
+
+What:		/sys/class/mic/mic(x)
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The directories /sys/class/mic/mic0, /sys/class/mic/mic1 etc.,
+		represent MIC devices (0,1,..etc). Each directory has
+		information specific to that MIC device.
+
+What:		/sys/class/mic/mic(x)/family
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the Coprocessor family for an Intel
+		MIC device. For example - "x100"
+
+What:		/sys/class/mic/mic(x)/stepping
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the silicon stepping for an Intel
+		MIC device. For example - "A0" or "B0"

drivers/misc/Kconfig +1 -0

index 8dacd4c..e760715 100644
@@ -537,4 +537,5 @@ source "drivers/misc/carma/Kconfig"
 source "drivers/misc/altera-stapl/Kconfig"
 source "drivers/misc/mei/Kconfig"
 source "drivers/misc/vmw_vmci/Kconfig"
+source "drivers/misc/mic/Kconfig"
 endmenu

drivers/misc/Makefile +1 -0

index c235d5b..0b7ea3e 100644
@@ -53,3 +53,4 @@ obj-$(CONFIG_INTEL_MEI)		+= mei/
 obj-$(CONFIG_VMWARE_VMCI)	+= vmw_vmci/
 obj-$(CONFIG_LATTICE_ECP3_CONFIG)	+= lattice-ecp3-config.o
 obj-$(CONFIG_SRAM)		+= sram.o
+obj-y				+= mic/

drivers/misc/mic/Kconfig new file +19 -0

new file mode 100644
index 0000000..aaefd0c
@@ -0,0 +1,19 @@
+comment "Intel MIC Host Driver"
+
+config INTEL_MIC_HOST
+	tristate "Intel MIC Host Driver"
+	depends on 64BIT && PCI
+	default N
+	help
+	  This enables Host Driver support for the Intel Many Integrated
+	  Core (MIC) family of PCIe form factor coprocessor devices that
+	  run a 64 bit Linux OS. The driver manages card OS state and
+	  enables communication between host and card. Intel MIC X100
+	  devices are currently supported.
+
+	  If you are building a host kernel with an Intel MIC device then
+	  say M (recommended) or Y, else say N. If unsure say N.
+
+	  More information about the Intel MIC family as well as the Linux
+	  OS and tools for MIC to use with this driver are available from
+	  <http://software.intel.com/en-us/mic-developer>.

drivers/misc/mic/Makefile new file +5 -0

new file mode 100644
index 0000000..8e72421
@@ -0,0 +1,5 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += host/

drivers/misc/mic/common/mic_device.h new file +37 -0

new file mode 100644
index 0000000..f02262e
@@ -0,0 +1,37 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC driver.
+ *
+ */
+#ifndef __MIC_COMMON_DEVICE_H_
+#define __MIC_COMMON_DEVICE_H_
+
+/**
+ * struct mic_mw - MIC memory window
+ *
+ * @pa: Base physical address.
+ * @va: Base ioremap'd virtual address.
+ * @len: Size of the memory window.
+ */
+struct mic_mw {
+	phys_addr_t pa;
+	void __iomem *va;
+	resource_size_t len;
+};
+
+#endif

drivers/misc/mic/host/Makefile new file +8 -0

new file mode 100644
index 0000000..93b9d25
@@ -0,0 +1,8 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += mic_host.o
+mic_host-objs := mic_main.o
+mic_host-objs += mic_x100.o
+mic_host-objs += mic_sysfs.o

drivers/misc/mic/host/mic_device.h new file +109 -0

109 lines not on this page · raw message

drivers/misc/mic/host/mic_main.c new file +309 -0

309 lines not on this page · raw message

drivers/misc/mic/host/mic_sysfs.c new file +97 -0

97 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.c new file +75 -0

75 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.h new file +47 -0

47 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:51 UTC #17 lore raw

PATCH v3 5/7 Intel MIC Host Driver Changes for Virtio Devices.

In reply to Sudeep Dutt

From: Ashutosh Dixit <[email protected]>

Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 drivers/misc/mic/Kconfig             |   1 +
 drivers/misc/mic/common/mic_device.h |   7 +
 drivers/misc/mic/host/Makefile       |   2 +
 drivers/misc/mic/host/mic_boot.c     |   3 +-
 drivers/misc/mic/host/mic_debugfs.c  | 140 +++++++
 drivers/misc/mic/host/mic_device.h   |   5 +
 drivers/misc/mic/host/mic_fops.c     | 221 +++++++++++
 drivers/misc/mic/host/mic_fops.h     |  32 ++
 drivers/misc/mic/host/mic_main.c     |  27 +-
 drivers/misc/mic/host/mic_virtio.c   | 704 +++++++++++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_virtio.h   | 138 +++++++
 include/uapi/linux/Kbuild            |   1 +
 include/uapi/linux/mic_common.h      | 166 ++++++++-
 include/uapi/linux/mic_ioctl.h       |  74 ++++
 14 files changed, 1518 insertions(+), 3 deletions(-)
 create mode 100644 drivers/misc/mic/host/mic_fops.c
 create mode 100644 drivers/misc/mic/host/mic_fops.h
 create mode 100644 drivers/misc/mic/host/mic_virtio.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.h
 create mode 100644 include/uapi/linux/mic_ioctl.h

drivers/misc/mic/Kconfig +1 -0

index 279a2e6..01f1a4a 100644
@@ -3,6 +3,7 @@ comment "Intel MIC Host Driver"
 config INTEL_MIC_HOST
 	tristate "Intel MIC Host Driver"
 	depends on 64BIT && PCI
+	select VHOST_RING
 	default N
 	help
 	  This enables Host Driver support for the Intel Many Integrated

drivers/misc/mic/common/mic_device.h +7 -0

index 6440e9d..01eb74f 100644
@@ -41,4 +41,11 @@ struct mic_mw {
 #define MIC_DPLO_SPAD 14
 #define MIC_DPHI_SPAD 15
 
+/*
+ * These values are supposed to be in the config_change field of the
+ * device page when the host sends a config change interrupt to the card.
+ */
+#define MIC_VIRTIO_PARAM_DEV_REMOVE 0x1
+#define MIC_VIRTIO_PARAM_CONFIG_CHANGED 0x2
+
 #endif

drivers/misc/mic/host/Makefile +2 -0

index 98bf565..e73ac74 100644
@@ -9,3 +9,5 @@ mic_host-objs += mic_sysfs.o
 mic_host-objs += mic_smpt.o
 mic_host-objs += mic_boot.o
 mic_host-objs += mic_debugfs.o
+mic_host-objs += mic_fops.o
+mic_host-objs += mic_virtio.o

drivers/misc/mic/host/mic_boot.c +2 -1

index 936fc58..fd9ff6d 100644
@@ -20,12 +20,12 @@
  */
 #include <linux/delay.h>
 #include <linux/firmware.h>
-#include <linux/interrupt.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_smpt.h"
+#include "mic_virtio.h"
 
 /**
  * mic_reset - Reset the MIC device.

@@ -117,6 +117,7 @@ void mic_stop(struct mic_device *mdev, bool force)
 {
 	mutex_lock(&mdev->mic_mutex);
 	if (MIC_OFFLINE != mdev->state || force) {
+		mic_virtio_reset_devices(mdev);
 		mic_bootparam_init(mdev);
 		mic_reset(mdev);
 		if (MIC_RESET_FAILED == mdev->state)

drivers/misc/mic/host/mic_debugfs.c +140 -0

171 lines not on this page · raw message

drivers/misc/mic/host/mic_device.h +5 -0

index c2ca811..4592e7d 100644
@@ -21,6 +21,7 @@
 #ifndef _MIC_DEVICE_H_
 #define _MIC_DEVICE_H_
 
+#include <linux/cdev.h>
 #include <linux/idr.h>
 
 /* The maximum number of MIC devices supported in a single host system. */

@@ -156,6 +157,8 @@ struct mic_irq;
  * @dp_dma_addr: virtio device page DMA address.
  * @shutdown_db: shutdown doorbell.
  * @shutdown_cookie: shutdown cookie.
+ * @cdev: Character device for MIC.
+ * @vdev_list: list of virtio devices.
  */
 struct mic_device {
 	struct mic_mw mmio;

@@ -189,6 +192,8 @@ struct mic_device {
 	dma_addr_t dp_dma_addr;
 	int shutdown_db;
 	struct mic_irq *shutdown_cookie;
+	struct cdev cdev;
+	struct list_head vdev_list;
 };
 
 /**

drivers/misc/mic/host/mic_fops.c new file +221 -0

221 lines not on this page · raw message

drivers/misc/mic/host/mic_fops.h new file +32 -0

new file mode 100644
index 0000000..dc3893d
@@ -0,0 +1,32 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Host driver.
+ *
+ */
+#ifndef _MIC_FOPS_H_
+#define _MIC_FOPS_H_
+
+int mic_open(struct inode *inode, struct file *filp);
+int mic_release(struct inode *inode, struct file *filp);
+ssize_t mic_read(struct file *filp, char __user *buf,
+			size_t count, loff_t *pos);
+long mic_ioctl(struct file *filp, unsigned int cmd, unsigned long arg);
+int mic_mmap(struct file *f, struct vm_area_struct *vma);
+unsigned int mic_poll(struct file *f, poll_table *wait);
+
+#endif

drivers/misc/mic/host/mic_main.c +26 -1

index 2f34a26..7c4f973 100644
@@ -22,16 +22,18 @@
  * 1) Enable DMA support.
  * 2) Enable per vring interrupt support.
  */
-#include <linux/fs.h>
 #include <linux/interrupt.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/poll.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_x100.h"
 #include "mic_smpt.h"
+#include "mic_fops.h"
+#include "mic_virtio.h"
 
 static const char mic_driver_name[] = "mic";

@@ -65,6 +67,15 @@ static struct class *g_mic_class;
 /* Base device node number for MIC devices */
 static dev_t g_mic_devno;
 
+static const struct file_operations mic_fops = {
+	.open = mic_open,
+	.release = mic_release,
+	.unlocked_ioctl = mic_ioctl,
+	.poll = mic_poll,
+	.mmap = mic_mmap,
+	.owner = THIS_MODULE,
+};
+
 /* Initialize the device page */
 static int mic_dp_init(struct mic_device *mdev)
 {

@@ -814,6 +825,7 @@ mic_device_init(struct mic_device *mdev, struct pci_dev *pdev)
 	mdev->irq_info.next_avail_src = 0;
 	INIT_WORK(&mdev->reset_trigger_work, mic_reset_trigger_work);
 	INIT_WORK(&mdev->shutdown_work, mic_shutdown_work);
+	INIT_LIST_HEAD(&mdev->vdev_list);
 }
 
 /**

@@ -951,7 +963,19 @@ static int mic_probe(struct pci_dev *pdev,
 	mic_bootparam_init(mdev);
 
 	mic_create_debug_dir(mdev);
+	cdev_init(&mdev->cdev, &mic_fops);
+	mdev->cdev.owner = THIS_MODULE;
+	rc = cdev_add(&mdev->cdev, MKDEV(MAJOR(g_mic_devno), mdev->id), 1);
+	if (rc) {
+		dev_err(&pdev->dev, "cdev_add err id %d rc %d\n", mdev->id, rc);
+		goto cleanup_debug_dir;
+	}
 	return 0;
+cleanup_debug_dir:
+	mic_delete_debug_dir(mdev);
+	mutex_lock(&mdev->mic_mutex);
+	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);
+	mutex_unlock(&mdev->mic_mutex);
 dp_uninit:
 	mic_dp_uninit(mdev);
 sysfs_put:

@@ -996,6 +1020,7 @@ static void mic_remove(struct pci_dev *pdev)
 		return;
 
 	mic_stop(mdev, false);
+	cdev_del(&mdev->cdev);
 	mic_delete_debug_dir(mdev);
 	mutex_lock(&mdev->mic_mutex);
 	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);

drivers/misc/mic/host/mic_virtio.c new file +704 -0

704 lines not on this page · raw message

drivers/misc/mic/host/mic_virtio.h new file +138 -0

138 lines not on this page · raw message

include/uapi/linux/Kbuild +1 -0

index e3f7b37..c73dbd0 100644
@@ -241,6 +241,7 @@ header-y += mei.h
 header-y += mempolicy.h
 header-y += meye.h
 header-y += mic_common.h
+header-y += mic_ioctl.h
 header-y += mii.h
 header-y += minix_fs.h
 header-y += mman.h

include/uapi/linux/mic_common.h +165 -1

178 lines not on this page · raw message

include/uapi/linux/mic_ioctl.h new file +74 -0

74 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:51 UTC #28 lore raw

PATCH v3 7/7 Sample Implementation of Intel MIC User Space Daemon.

In reply to Sudeep Dutt

From: Caz Yokoyama <[email protected]>

Co-author: Ashutosh Dixit <[email protected]>
Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

---
 Documentation/mic/mic_overview.txt |   49 ++
 Documentation/mic/mpssd/.gitignore |    1 +
 Documentation/mic/mpssd/Makefile   |   19 +
 Documentation/mic/mpssd/micctrl    |  173 ++++
 Documentation/mic/mpssd/mpss       |  202 +++++
 Documentation/mic/mpssd/mpssd.c    | 1701 ++++++++++++++++++++++++++++++++++++
 Documentation/mic/mpssd/mpssd.h    |  100 +++
 Documentation/mic/mpssd/sysfs.c    |  102 +++
 8 files changed, 2347 insertions(+)
 create mode 100644 Documentation/mic/mic_overview.txt
 create mode 100644 Documentation/mic/mpssd/.gitignore
 create mode 100644 Documentation/mic/mpssd/Makefile
 create mode 100755 Documentation/mic/mpssd/micctrl
 create mode 100755 Documentation/mic/mpssd/mpss
 create mode 100644 Documentation/mic/mpssd/mpssd.c
 create mode 100644 Documentation/mic/mpssd/mpssd.h
 create mode 100644 Documentation/mic/mpssd/sysfs.c

Documentation/mic/mic_overview.txt new file +49 -0

new file mode 100644
index 0000000..c4424ed
@@ -0,0 +1,49 @@
+An Intel MIC X100 device is a PCIe form factor add-in coprocessor
+card based on the Intel Many Integrated Core (MIC) architecture
+that runs a Linux OS. It is a PCIe endpoint in a platform and therefore
+implements the three required standard address spaces i.e. configuration,
+memory and I/O. The host OS loads a device driver as is typical for
+PCIe devices. The card itself runs a bootstrap after reset that
+transfers control to the card OS downloaded from the host driver.
+The card OS as shipped by Intel is a Linux kernel with modifications
+for the X100 devices.
+
+Since it is a PCIe card, it does not have the ability to host hardware
+devices for networking, storage and console. We provide these devices
+on X100 coprocessors thus enabling a self-bootable equivalent environment
+for applications. A key benefit of our solution is that it leverages
+the standard virtio framework for network, disk and console devices,
+though in our case the virtio framework is used across a PCIe bus.
+
+Here is a block diagram of the various components described above. The
+virtio backends are situated on the host rather than the card given better
+single threaded performance for the host compared to MIC, the ability of
+the host to initiate DMA's to/from the card using the MIC DMA engine and
+the fact that the virtio block storage backend can only be on the host.
+
+                              |
+       +----------+           |             +----------+
+       | Card OS  |           |             | Host OS  |
+       +----------+           |             +----------+
+                              |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+| Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
+| Net   | |Console | |Block | | |Net      |  |Console | |Block   |
+| Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+    |         |         |     |      |            |         |
+    |         |         |     |User  |            |         |
+    |         |         |     |------|------------|---------|-------
+    +-------------------+     |Kernel +--------------------------+
+              |               |       | Virtio over PCIe IOCTLs  |
+              |               |       +--------------------------+
+      +--------------+        |                   |
+      |Intel MIC     |        |            +---------------+
+      |Card Driver   |        |            |Intel MIC      |
+      +--------------+        |            |Host Driver    |
+              |               |            +---------------+
+              |               |                   |
+     +-------------------------------------------------------------+
+     |                                                             |
+     |                    PCIe Bus                                 |
+     +-------------------------------------------------------------+

Documentation/mic/mpssd/.gitignore new file +1 -0

new file mode 100644
index 0000000..8b7c72f
@@ -0,0 +1 @@
+mpssd

Documentation/mic/mpssd/Makefile new file +19 -0

new file mode 100644
index 0000000..eb860a7
@@ -0,0 +1,19 @@
+#
+# Makefile - Intel MIC User Space Tools.
+# Copyright(c) 2013, Intel Corporation.
+#
+ifdef DEBUG
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall -DDEBUG=$(DEBUG)
+else
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall
+endif
+
+mpssd: mpssd.o sysfs.o
+	$(CC) $(CFLAGS) -o $@ $^ -lpthread
+
+install:
+	install mpssd /usr/sbin/mpssd
+	install micctrl /usr/sbin/micctrl
+
+clean:
+	rm -f mpssd *.o

Documentation/mic/mpssd/micctrl new file +173 -0

173 lines not on this page · raw message

Documentation/mic/mpssd/mpss new file +202 -0

202 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.c new file +1701 -0

1,701 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.h new file +100 -0

100 lines not on this page · raw message

Documentation/mic/mpssd/sysfs.c new file +102 -0

102 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:51 UTC #7 lore raw

PATCH v3 1/7 Intel MIC Host Driver for X100 family.

In reply to Sudeep Dutt

This patch enables the following:
a) Initializes the Intel MIC X100 PCIe devices.
b) Provides sysfs entries for family and stepping information.

Co-author: Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 Documentation/ABI/testing/sysfs-class-mic.txt |  34 +++
 drivers/misc/Kconfig                          |   1 +
 drivers/misc/Makefile                         |   1 +
 drivers/misc/mic/Kconfig                      |  19 ++
 drivers/misc/mic/Makefile                     |   5 +
 drivers/misc/mic/common/mic_device.h          |  37 +++
 drivers/misc/mic/host/Makefile                |   8 +
 drivers/misc/mic/host/mic_device.h            | 109 +++++++++
 drivers/misc/mic/host/mic_main.c              | 309 ++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_sysfs.c             |  97 ++++++++
 drivers/misc/mic/host/mic_x100.c              |  75 +++++++
 drivers/misc/mic/host/mic_x100.h              |  47 ++++
 12 files changed, 742 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt
 create mode 100644 drivers/misc/mic/Kconfig
 create mode 100644 drivers/misc/mic/Makefile
 create mode 100644 drivers/misc/mic/common/mic_device.h
 create mode 100644 drivers/misc/mic/host/Makefile
 create mode 100644 drivers/misc/mic/host/mic_device.h
 create mode 100644 drivers/misc/mic/host/mic_main.c
 create mode 100644 drivers/misc/mic/host/mic_sysfs.c
 create mode 100644 drivers/misc/mic/host/mic_x100.c
 create mode 100644 drivers/misc/mic/host/mic_x100.h

Documentation/ABI/testing/sysfs-class-mic.txt new file +34 -0

new file mode 100644
index 0000000..09eb3c6
@@ -0,0 +1,34 @@
+What:		/sys/class/mic/
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The mic class directory belongs to Intel MIC devices and
+		provides information per MIC device. An Intel MIC device is a
+		PCIe form factor add-in Coprocessor card based on the Intel Many
+		Integrated Core (MIC) architecture that runs a Linux OS.
+
+What:		/sys/class/mic/mic(x)
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The directories /sys/class/mic/mic0, /sys/class/mic/mic1 etc.,
+		represent MIC devices (0,1,..etc). Each directory has
+		information specific to that MIC device.
+
+What:		/sys/class/mic/mic(x)/family
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the Coprocessor family for an Intel
+		MIC device. For example - "x100"
+
+What:		/sys/class/mic/mic(x)/stepping
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the silicon stepping for an Intel
+		MIC device. For example - "A0" or "B0"

drivers/misc/Kconfig +1 -0

index 8dacd4c..e760715 100644
@@ -537,4 +537,5 @@ source "drivers/misc/carma/Kconfig"
 source "drivers/misc/altera-stapl/Kconfig"
 source "drivers/misc/mei/Kconfig"
 source "drivers/misc/vmw_vmci/Kconfig"
+source "drivers/misc/mic/Kconfig"
 endmenu

drivers/misc/Makefile +1 -0

index c235d5b..0b7ea3e 100644
@@ -53,3 +53,4 @@ obj-$(CONFIG_INTEL_MEI)		+= mei/
 obj-$(CONFIG_VMWARE_VMCI)	+= vmw_vmci/
 obj-$(CONFIG_LATTICE_ECP3_CONFIG)	+= lattice-ecp3-config.o
 obj-$(CONFIG_SRAM)		+= sram.o
+obj-y				+= mic/

drivers/misc/mic/Kconfig new file +19 -0

new file mode 100644
index 0000000..aaefd0c
@@ -0,0 +1,19 @@
+comment "Intel MIC Host Driver"
+
+config INTEL_MIC_HOST
+	tristate "Intel MIC Host Driver"
+	depends on 64BIT && PCI
+	default N
+	help
+	  This enables Host Driver support for the Intel Many Integrated
+	  Core (MIC) family of PCIe form factor coprocessor devices that
+	  run a 64 bit Linux OS. The driver manages card OS state and
+	  enables communication between host and card. Intel MIC X100
+	  devices are currently supported.
+
+	  If you are building a host kernel with an Intel MIC device then
+	  say M (recommended) or Y, else say N. If unsure say N.
+
+	  More information about the Intel MIC family as well as the Linux
+	  OS and tools for MIC to use with this driver are available from
+	  <http://software.intel.com/en-us/mic-developer>.

drivers/misc/mic/Makefile new file +5 -0

new file mode 100644
index 0000000..8e72421
@@ -0,0 +1,5 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += host/

drivers/misc/mic/common/mic_device.h new file +37 -0

new file mode 100644
index 0000000..f02262e
@@ -0,0 +1,37 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC driver.
+ *
+ */
+#ifndef __MIC_COMMON_DEVICE_H_
+#define __MIC_COMMON_DEVICE_H_
+
+/**
+ * struct mic_mw - MIC memory window
+ *
+ * @pa: Base physical address.
+ * @va: Base ioremap'd virtual address.
+ * @len: Size of the memory window.
+ */
+struct mic_mw {
+	phys_addr_t pa;
+	void __iomem *va;
+	resource_size_t len;
+};
+
+#endif

drivers/misc/mic/host/Makefile new file +8 -0

new file mode 100644
index 0000000..93b9d25
@@ -0,0 +1,8 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += mic_host.o
+mic_host-objs := mic_main.o
+mic_host-objs += mic_x100.o
+mic_host-objs += mic_sysfs.o

drivers/misc/mic/host/mic_device.h new file +109 -0

109 lines not on this page · raw message

drivers/misc/mic/host/mic_main.c new file +309 -0

309 lines not on this page · raw message

drivers/misc/mic/host/mic_sysfs.c new file +97 -0

97 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.c new file +75 -0

75 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.h new file +47 -0

47 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:52 UTC #18 lore raw

PATCH v3 5/7 Intel MIC Host Driver Changes for Virtio Devices.

In reply to Sudeep Dutt

From: Ashutosh Dixit <[email protected]>

Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 drivers/misc/mic/Kconfig             |   1 +
 drivers/misc/mic/common/mic_device.h |   7 +
 drivers/misc/mic/host/Makefile       |   2 +
 drivers/misc/mic/host/mic_boot.c     |   3 +-
 drivers/misc/mic/host/mic_debugfs.c  | 140 +++++++
 drivers/misc/mic/host/mic_device.h   |   5 +
 drivers/misc/mic/host/mic_fops.c     | 221 +++++++++++
 drivers/misc/mic/host/mic_fops.h     |  32 ++
 drivers/misc/mic/host/mic_main.c     |  27 +-
 drivers/misc/mic/host/mic_virtio.c   | 704 +++++++++++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_virtio.h   | 138 +++++++
 include/uapi/linux/Kbuild            |   1 +
 include/uapi/linux/mic_common.h      | 166 ++++++++-
 include/uapi/linux/mic_ioctl.h       |  74 ++++
 14 files changed, 1518 insertions(+), 3 deletions(-)
 create mode 100644 drivers/misc/mic/host/mic_fops.c
 create mode 100644 drivers/misc/mic/host/mic_fops.h
 create mode 100644 drivers/misc/mic/host/mic_virtio.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.h
 create mode 100644 include/uapi/linux/mic_ioctl.h

drivers/misc/mic/Kconfig +1 -0

index 279a2e6..01f1a4a 100644
@@ -3,6 +3,7 @@ comment "Intel MIC Host Driver"
 config INTEL_MIC_HOST
 	tristate "Intel MIC Host Driver"
 	depends on 64BIT && PCI
+	select VHOST_RING
 	default N
 	help
 	  This enables Host Driver support for the Intel Many Integrated

drivers/misc/mic/common/mic_device.h +7 -0

index 6440e9d..01eb74f 100644
@@ -41,4 +41,11 @@ struct mic_mw {
 #define MIC_DPLO_SPAD 14
 #define MIC_DPHI_SPAD 15
 
+/*
+ * These values are supposed to be in the config_change field of the
+ * device page when the host sends a config change interrupt to the card.
+ */
+#define MIC_VIRTIO_PARAM_DEV_REMOVE 0x1
+#define MIC_VIRTIO_PARAM_CONFIG_CHANGED 0x2
+
 #endif

drivers/misc/mic/host/Makefile +2 -0

index 98bf565..e73ac74 100644
@@ -9,3 +9,5 @@ mic_host-objs += mic_sysfs.o
 mic_host-objs += mic_smpt.o
 mic_host-objs += mic_boot.o
 mic_host-objs += mic_debugfs.o
+mic_host-objs += mic_fops.o
+mic_host-objs += mic_virtio.o

drivers/misc/mic/host/mic_boot.c +2 -1

index 936fc58..fd9ff6d 100644
@@ -20,12 +20,12 @@
  */
 #include <linux/delay.h>
 #include <linux/firmware.h>
-#include <linux/interrupt.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_smpt.h"
+#include "mic_virtio.h"
 
 /**
  * mic_reset - Reset the MIC device.

@@ -117,6 +117,7 @@ void mic_stop(struct mic_device *mdev, bool force)
 {
 	mutex_lock(&mdev->mic_mutex);
 	if (MIC_OFFLINE != mdev->state || force) {
+		mic_virtio_reset_devices(mdev);
 		mic_bootparam_init(mdev);
 		mic_reset(mdev);
 		if (MIC_RESET_FAILED == mdev->state)

drivers/misc/mic/host/mic_debugfs.c +140 -0

171 lines not on this page · raw message

drivers/misc/mic/host/mic_device.h +5 -0

index c2ca811..4592e7d 100644
@@ -21,6 +21,7 @@
 #ifndef _MIC_DEVICE_H_
 #define _MIC_DEVICE_H_
 
+#include <linux/cdev.h>
 #include <linux/idr.h>
 
 /* The maximum number of MIC devices supported in a single host system. */

@@ -156,6 +157,8 @@ struct mic_irq;
  * @dp_dma_addr: virtio device page DMA address.
  * @shutdown_db: shutdown doorbell.
  * @shutdown_cookie: shutdown cookie.
+ * @cdev: Character device for MIC.
+ * @vdev_list: list of virtio devices.
  */
 struct mic_device {
 	struct mic_mw mmio;

@@ -189,6 +192,8 @@ struct mic_device {
 	dma_addr_t dp_dma_addr;
 	int shutdown_db;
 	struct mic_irq *shutdown_cookie;
+	struct cdev cdev;
+	struct list_head vdev_list;
 };
 
 /**

drivers/misc/mic/host/mic_fops.c new file +221 -0

221 lines not on this page · raw message

drivers/misc/mic/host/mic_fops.h new file +32 -0

new file mode 100644
index 0000000..dc3893d
@@ -0,0 +1,32 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Host driver.
+ *
+ */
+#ifndef _MIC_FOPS_H_
+#define _MIC_FOPS_H_
+
+int mic_open(struct inode *inode, struct file *filp);
+int mic_release(struct inode *inode, struct file *filp);
+ssize_t mic_read(struct file *filp, char __user *buf,
+			size_t count, loff_t *pos);
+long mic_ioctl(struct file *filp, unsigned int cmd, unsigned long arg);
+int mic_mmap(struct file *f, struct vm_area_struct *vma);
+unsigned int mic_poll(struct file *f, poll_table *wait);
+
+#endif

drivers/misc/mic/host/mic_main.c +26 -1

index 2f34a26..7c4f973 100644
@@ -22,16 +22,18 @@
  * 1) Enable DMA support.
  * 2) Enable per vring interrupt support.
  */
-#include <linux/fs.h>
 #include <linux/interrupt.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/poll.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_x100.h"
 #include "mic_smpt.h"
+#include "mic_fops.h"
+#include "mic_virtio.h"
 
 static const char mic_driver_name[] = "mic";

@@ -65,6 +67,15 @@ static struct class *g_mic_class;
 /* Base device node number for MIC devices */
 static dev_t g_mic_devno;
 
+static const struct file_operations mic_fops = {
+	.open = mic_open,
+	.release = mic_release,
+	.unlocked_ioctl = mic_ioctl,
+	.poll = mic_poll,
+	.mmap = mic_mmap,
+	.owner = THIS_MODULE,
+};
+
 /* Initialize the device page */
 static int mic_dp_init(struct mic_device *mdev)
 {

@@ -814,6 +825,7 @@ mic_device_init(struct mic_device *mdev, struct pci_dev *pdev)
 	mdev->irq_info.next_avail_src = 0;
 	INIT_WORK(&mdev->reset_trigger_work, mic_reset_trigger_work);
 	INIT_WORK(&mdev->shutdown_work, mic_shutdown_work);
+	INIT_LIST_HEAD(&mdev->vdev_list);
 }
 
 /**

@@ -951,7 +963,19 @@ static int mic_probe(struct pci_dev *pdev,
 	mic_bootparam_init(mdev);
 
 	mic_create_debug_dir(mdev);
+	cdev_init(&mdev->cdev, &mic_fops);
+	mdev->cdev.owner = THIS_MODULE;
+	rc = cdev_add(&mdev->cdev, MKDEV(MAJOR(g_mic_devno), mdev->id), 1);
+	if (rc) {
+		dev_err(&pdev->dev, "cdev_add err id %d rc %d\n", mdev->id, rc);
+		goto cleanup_debug_dir;
+	}
 	return 0;
+cleanup_debug_dir:
+	mic_delete_debug_dir(mdev);
+	mutex_lock(&mdev->mic_mutex);
+	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);
+	mutex_unlock(&mdev->mic_mutex);
 dp_uninit:
 	mic_dp_uninit(mdev);
 sysfs_put:

@@ -996,6 +1020,7 @@ static void mic_remove(struct pci_dev *pdev)
 		return;
 
 	mic_stop(mdev, false);
+	cdev_del(&mdev->cdev);
 	mic_delete_debug_dir(mdev);
 	mutex_lock(&mdev->mic_mutex);
 	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);

drivers/misc/mic/host/mic_virtio.c new file +704 -0

704 lines not on this page · raw message

drivers/misc/mic/host/mic_virtio.h new file +138 -0

138 lines not on this page · raw message

include/uapi/linux/Kbuild +1 -0

index e3f7b37..c73dbd0 100644
@@ -241,6 +241,7 @@ header-y += mei.h
 header-y += mempolicy.h
 header-y += meye.h
 header-y += mic_common.h
+header-y += mic_ioctl.h
 header-y += mii.h
 header-y += minix_fs.h
 header-y += mman.h

include/uapi/linux/mic_common.h +165 -1

178 lines not on this page · raw message

include/uapi/linux/mic_ioctl.h new file +74 -0

74 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:52 UTC #29 lore raw

PATCH v3 7/7 Sample Implementation of Intel MIC User Space Daemon.

In reply to Sudeep Dutt

From: Caz Yokoyama <[email protected]>

Co-author: Ashutosh Dixit <[email protected]>
Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

---
 Documentation/mic/mic_overview.txt |   49 ++
 Documentation/mic/mpssd/.gitignore |    1 +
 Documentation/mic/mpssd/Makefile   |   19 +
 Documentation/mic/mpssd/micctrl    |  173 ++++
 Documentation/mic/mpssd/mpss       |  202 +++++
 Documentation/mic/mpssd/mpssd.c    | 1701 ++++++++++++++++++++++++++++++++++++
 Documentation/mic/mpssd/mpssd.h    |  100 +++
 Documentation/mic/mpssd/sysfs.c    |  102 +++
 8 files changed, 2347 insertions(+)
 create mode 100644 Documentation/mic/mic_overview.txt
 create mode 100644 Documentation/mic/mpssd/.gitignore
 create mode 100644 Documentation/mic/mpssd/Makefile
 create mode 100755 Documentation/mic/mpssd/micctrl
 create mode 100755 Documentation/mic/mpssd/mpss
 create mode 100644 Documentation/mic/mpssd/mpssd.c
 create mode 100644 Documentation/mic/mpssd/mpssd.h
 create mode 100644 Documentation/mic/mpssd/sysfs.c

Documentation/mic/mic_overview.txt new file +49 -0

new file mode 100644
index 0000000..c4424ed
@@ -0,0 +1,49 @@
+An Intel MIC X100 device is a PCIe form factor add-in coprocessor
+card based on the Intel Many Integrated Core (MIC) architecture
+that runs a Linux OS. It is a PCIe endpoint in a platform and therefore
+implements the three required standard address spaces i.e. configuration,
+memory and I/O. The host OS loads a device driver as is typical for
+PCIe devices. The card itself runs a bootstrap after reset that
+transfers control to the card OS downloaded from the host driver.
+The card OS as shipped by Intel is a Linux kernel with modifications
+for the X100 devices.
+
+Since it is a PCIe card, it does not have the ability to host hardware
+devices for networking, storage and console. We provide these devices
+on X100 coprocessors thus enabling a self-bootable equivalent environment
+for applications. A key benefit of our solution is that it leverages
+the standard virtio framework for network, disk and console devices,
+though in our case the virtio framework is used across a PCIe bus.
+
+Here is a block diagram of the various components described above. The
+virtio backends are situated on the host rather than the card given better
+single threaded performance for the host compared to MIC, the ability of
+the host to initiate DMA's to/from the card using the MIC DMA engine and
+the fact that the virtio block storage backend can only be on the host.
+
+                              |
+       +----------+           |             +----------+
+       | Card OS  |           |             | Host OS  |
+       +----------+           |             +----------+
+                              |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+| Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
+| Net   | |Console | |Block | | |Net      |  |Console | |Block   |
+| Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+    |         |         |     |      |            |         |
+    |         |         |     |User  |            |         |
+    |         |         |     |------|------------|---------|-------
+    +-------------------+     |Kernel +--------------------------+
+              |               |       | Virtio over PCIe IOCTLs  |
+              |               |       +--------------------------+
+      +--------------+        |                   |
+      |Intel MIC     |        |            +---------------+
+      |Card Driver   |        |            |Intel MIC      |
+      +--------------+        |            |Host Driver    |
+              |               |            +---------------+
+              |               |                   |
+     +-------------------------------------------------------------+
+     |                                                             |
+     |                    PCIe Bus                                 |
+     +-------------------------------------------------------------+

Documentation/mic/mpssd/.gitignore new file +1 -0

new file mode 100644
index 0000000..8b7c72f
@@ -0,0 +1 @@
+mpssd

Documentation/mic/mpssd/Makefile new file +19 -0

new file mode 100644
index 0000000..eb860a7
@@ -0,0 +1,19 @@
+#
+# Makefile - Intel MIC User Space Tools.
+# Copyright(c) 2013, Intel Corporation.
+#
+ifdef DEBUG
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall -DDEBUG=$(DEBUG)
+else
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall
+endif
+
+mpssd: mpssd.o sysfs.o
+	$(CC) $(CFLAGS) -o $@ $^ -lpthread
+
+install:
+	install mpssd /usr/sbin/mpssd
+	install micctrl /usr/sbin/micctrl
+
+clean:
+	rm -f mpssd *.o

Documentation/mic/mpssd/micctrl new file +173 -0

173 lines not on this page · raw message

Documentation/mic/mpssd/mpss new file +202 -0

202 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.c new file +1701 -0

1,701 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.h new file +100 -0

100 lines not on this page · raw message

Documentation/mic/mpssd/sysfs.c new file +102 -0

102 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:52 UTC #8 lore raw

PATCH v3 1/7 Intel MIC Host Driver for X100 family.

In reply to Sudeep Dutt

This patch enables the following:
a) Initializes the Intel MIC X100 PCIe devices.
b) Provides sysfs entries for family and stepping information.

Co-author: Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 Documentation/ABI/testing/sysfs-class-mic.txt |  34 +++
 drivers/misc/Kconfig                          |   1 +
 drivers/misc/Makefile                         |   1 +
 drivers/misc/mic/Kconfig                      |  19 ++
 drivers/misc/mic/Makefile                     |   5 +
 drivers/misc/mic/common/mic_device.h          |  37 +++
 drivers/misc/mic/host/Makefile                |   8 +
 drivers/misc/mic/host/mic_device.h            | 109 +++++++++
 drivers/misc/mic/host/mic_main.c              | 309 ++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_sysfs.c             |  97 ++++++++
 drivers/misc/mic/host/mic_x100.c              |  75 +++++++
 drivers/misc/mic/host/mic_x100.h              |  47 ++++
 12 files changed, 742 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt
 create mode 100644 drivers/misc/mic/Kconfig
 create mode 100644 drivers/misc/mic/Makefile
 create mode 100644 drivers/misc/mic/common/mic_device.h
 create mode 100644 drivers/misc/mic/host/Makefile
 create mode 100644 drivers/misc/mic/host/mic_device.h
 create mode 100644 drivers/misc/mic/host/mic_main.c
 create mode 100644 drivers/misc/mic/host/mic_sysfs.c
 create mode 100644 drivers/misc/mic/host/mic_x100.c
 create mode 100644 drivers/misc/mic/host/mic_x100.h

Documentation/ABI/testing/sysfs-class-mic.txt new file +34 -0

new file mode 100644
index 0000000..09eb3c6
@@ -0,0 +1,34 @@
+What:		/sys/class/mic/
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The mic class directory belongs to Intel MIC devices and
+		provides information per MIC device. An Intel MIC device is a
+		PCIe form factor add-in Coprocessor card based on the Intel Many
+		Integrated Core (MIC) architecture that runs a Linux OS.
+
+What:		/sys/class/mic/mic(x)
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The directories /sys/class/mic/mic0, /sys/class/mic/mic1 etc.,
+		represent MIC devices (0,1,..etc). Each directory has
+		information specific to that MIC device.
+
+What:		/sys/class/mic/mic(x)/family
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the Coprocessor family for an Intel
+		MIC device. For example - "x100"
+
+What:		/sys/class/mic/mic(x)/stepping
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the silicon stepping for an Intel
+		MIC device. For example - "A0" or "B0"

drivers/misc/Kconfig +1 -0

index 8dacd4c..e760715 100644
@@ -537,4 +537,5 @@ source "drivers/misc/carma/Kconfig"
 source "drivers/misc/altera-stapl/Kconfig"
 source "drivers/misc/mei/Kconfig"
 source "drivers/misc/vmw_vmci/Kconfig"
+source "drivers/misc/mic/Kconfig"
 endmenu

drivers/misc/Makefile +1 -0

index c235d5b..0b7ea3e 100644
@@ -53,3 +53,4 @@ obj-$(CONFIG_INTEL_MEI)		+= mei/
 obj-$(CONFIG_VMWARE_VMCI)	+= vmw_vmci/
 obj-$(CONFIG_LATTICE_ECP3_CONFIG)	+= lattice-ecp3-config.o
 obj-$(CONFIG_SRAM)		+= sram.o
+obj-y				+= mic/

drivers/misc/mic/Kconfig new file +19 -0

new file mode 100644
index 0000000..aaefd0c
@@ -0,0 +1,19 @@
+comment "Intel MIC Host Driver"
+
+config INTEL_MIC_HOST
+	tristate "Intel MIC Host Driver"
+	depends on 64BIT && PCI
+	default N
+	help
+	  This enables Host Driver support for the Intel Many Integrated
+	  Core (MIC) family of PCIe form factor coprocessor devices that
+	  run a 64 bit Linux OS. The driver manages card OS state and
+	  enables communication between host and card. Intel MIC X100
+	  devices are currently supported.
+
+	  If you are building a host kernel with an Intel MIC device then
+	  say M (recommended) or Y, else say N. If unsure say N.
+
+	  More information about the Intel MIC family as well as the Linux
+	  OS and tools for MIC to use with this driver are available from
+	  <http://software.intel.com/en-us/mic-developer>.

drivers/misc/mic/Makefile new file +5 -0

new file mode 100644
index 0000000..8e72421
@@ -0,0 +1,5 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += host/

drivers/misc/mic/common/mic_device.h new file +37 -0

new file mode 100644
index 0000000..f02262e
@@ -0,0 +1,37 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC driver.
+ *
+ */
+#ifndef __MIC_COMMON_DEVICE_H_
+#define __MIC_COMMON_DEVICE_H_
+
+/**
+ * struct mic_mw - MIC memory window
+ *
+ * @pa: Base physical address.
+ * @va: Base ioremap'd virtual address.
+ * @len: Size of the memory window.
+ */
+struct mic_mw {
+	phys_addr_t pa;
+	void __iomem *va;
+	resource_size_t len;
+};
+
+#endif

drivers/misc/mic/host/Makefile new file +8 -0

new file mode 100644
index 0000000..93b9d25
@@ -0,0 +1,8 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += mic_host.o
+mic_host-objs := mic_main.o
+mic_host-objs += mic_x100.o
+mic_host-objs += mic_sysfs.o

drivers/misc/mic/host/mic_device.h new file +109 -0

109 lines not on this page · raw message

drivers/misc/mic/host/mic_main.c new file +309 -0

309 lines not on this page · raw message

drivers/misc/mic/host/mic_sysfs.c new file +97 -0

97 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.c new file +75 -0

75 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.h new file +47 -0

47 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:52 UTC #19 lore raw

PATCH v3 5/7 Intel MIC Host Driver Changes for Virtio Devices.

In reply to Sudeep Dutt

From: Ashutosh Dixit <[email protected]>

Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 drivers/misc/mic/Kconfig             |   1 +
 drivers/misc/mic/common/mic_device.h |   7 +
 drivers/misc/mic/host/Makefile       |   2 +
 drivers/misc/mic/host/mic_boot.c     |   3 +-
 drivers/misc/mic/host/mic_debugfs.c  | 140 +++++++
 drivers/misc/mic/host/mic_device.h   |   5 +
 drivers/misc/mic/host/mic_fops.c     | 221 +++++++++++
 drivers/misc/mic/host/mic_fops.h     |  32 ++
 drivers/misc/mic/host/mic_main.c     |  27 +-
 drivers/misc/mic/host/mic_virtio.c   | 704 +++++++++++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_virtio.h   | 138 +++++++
 include/uapi/linux/Kbuild            |   1 +
 include/uapi/linux/mic_common.h      | 166 ++++++++-
 include/uapi/linux/mic_ioctl.h       |  74 ++++
 14 files changed, 1518 insertions(+), 3 deletions(-)
 create mode 100644 drivers/misc/mic/host/mic_fops.c
 create mode 100644 drivers/misc/mic/host/mic_fops.h
 create mode 100644 drivers/misc/mic/host/mic_virtio.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.h
 create mode 100644 include/uapi/linux/mic_ioctl.h

drivers/misc/mic/Kconfig +1 -0

index 279a2e6..01f1a4a 100644
@@ -3,6 +3,7 @@ comment "Intel MIC Host Driver"
 config INTEL_MIC_HOST
 	tristate "Intel MIC Host Driver"
 	depends on 64BIT && PCI
+	select VHOST_RING
 	default N
 	help
 	  This enables Host Driver support for the Intel Many Integrated

drivers/misc/mic/common/mic_device.h +7 -0

index 6440e9d..01eb74f 100644
@@ -41,4 +41,11 @@ struct mic_mw {
 #define MIC_DPLO_SPAD 14
 #define MIC_DPHI_SPAD 15
 
+/*
+ * These values are supposed to be in the config_change field of the
+ * device page when the host sends a config change interrupt to the card.
+ */
+#define MIC_VIRTIO_PARAM_DEV_REMOVE 0x1
+#define MIC_VIRTIO_PARAM_CONFIG_CHANGED 0x2
+
 #endif

drivers/misc/mic/host/Makefile +2 -0

index 98bf565..e73ac74 100644
@@ -9,3 +9,5 @@ mic_host-objs += mic_sysfs.o
 mic_host-objs += mic_smpt.o
 mic_host-objs += mic_boot.o
 mic_host-objs += mic_debugfs.o
+mic_host-objs += mic_fops.o
+mic_host-objs += mic_virtio.o

drivers/misc/mic/host/mic_boot.c +2 -1

index 936fc58..fd9ff6d 100644
@@ -20,12 +20,12 @@
  */
 #include <linux/delay.h>
 #include <linux/firmware.h>
-#include <linux/interrupt.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_smpt.h"
+#include "mic_virtio.h"
 
 /**
  * mic_reset - Reset the MIC device.

@@ -117,6 +117,7 @@ void mic_stop(struct mic_device *mdev, bool force)
 {
 	mutex_lock(&mdev->mic_mutex);
 	if (MIC_OFFLINE != mdev->state || force) {
+		mic_virtio_reset_devices(mdev);
 		mic_bootparam_init(mdev);
 		mic_reset(mdev);
 		if (MIC_RESET_FAILED == mdev->state)

drivers/misc/mic/host/mic_debugfs.c +140 -0

171 lines not on this page · raw message

drivers/misc/mic/host/mic_device.h +5 -0

index c2ca811..4592e7d 100644
@@ -21,6 +21,7 @@
 #ifndef _MIC_DEVICE_H_
 #define _MIC_DEVICE_H_
 
+#include <linux/cdev.h>
 #include <linux/idr.h>
 
 /* The maximum number of MIC devices supported in a single host system. */

@@ -156,6 +157,8 @@ struct mic_irq;
  * @dp_dma_addr: virtio device page DMA address.
  * @shutdown_db: shutdown doorbell.
  * @shutdown_cookie: shutdown cookie.
+ * @cdev: Character device for MIC.
+ * @vdev_list: list of virtio devices.
  */
 struct mic_device {
 	struct mic_mw mmio;

@@ -189,6 +192,8 @@ struct mic_device {
 	dma_addr_t dp_dma_addr;
 	int shutdown_db;
 	struct mic_irq *shutdown_cookie;
+	struct cdev cdev;
+	struct list_head vdev_list;
 };
 
 /**

drivers/misc/mic/host/mic_fops.c new file +221 -0

221 lines not on this page · raw message

drivers/misc/mic/host/mic_fops.h new file +32 -0

new file mode 100644
index 0000000..dc3893d
@@ -0,0 +1,32 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Host driver.
+ *
+ */
+#ifndef _MIC_FOPS_H_
+#define _MIC_FOPS_H_
+
+int mic_open(struct inode *inode, struct file *filp);
+int mic_release(struct inode *inode, struct file *filp);
+ssize_t mic_read(struct file *filp, char __user *buf,
+			size_t count, loff_t *pos);
+long mic_ioctl(struct file *filp, unsigned int cmd, unsigned long arg);
+int mic_mmap(struct file *f, struct vm_area_struct *vma);
+unsigned int mic_poll(struct file *f, poll_table *wait);
+
+#endif

drivers/misc/mic/host/mic_main.c +26 -1

index 2f34a26..7c4f973 100644
@@ -22,16 +22,18 @@
  * 1) Enable DMA support.
  * 2) Enable per vring interrupt support.
  */
-#include <linux/fs.h>
 #include <linux/interrupt.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/poll.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_x100.h"
 #include "mic_smpt.h"
+#include "mic_fops.h"
+#include "mic_virtio.h"
 
 static const char mic_driver_name[] = "mic";

@@ -65,6 +67,15 @@ static struct class *g_mic_class;
 /* Base device node number for MIC devices */
 static dev_t g_mic_devno;
 
+static const struct file_operations mic_fops = {
+	.open = mic_open,
+	.release = mic_release,
+	.unlocked_ioctl = mic_ioctl,
+	.poll = mic_poll,
+	.mmap = mic_mmap,
+	.owner = THIS_MODULE,
+};
+
 /* Initialize the device page */
 static int mic_dp_init(struct mic_device *mdev)
 {

@@ -814,6 +825,7 @@ mic_device_init(struct mic_device *mdev, struct pci_dev *pdev)
 	mdev->irq_info.next_avail_src = 0;
 	INIT_WORK(&mdev->reset_trigger_work, mic_reset_trigger_work);
 	INIT_WORK(&mdev->shutdown_work, mic_shutdown_work);
+	INIT_LIST_HEAD(&mdev->vdev_list);
 }
 
 /**

@@ -951,7 +963,19 @@ static int mic_probe(struct pci_dev *pdev,
 	mic_bootparam_init(mdev);
 
 	mic_create_debug_dir(mdev);
+	cdev_init(&mdev->cdev, &mic_fops);
+	mdev->cdev.owner = THIS_MODULE;
+	rc = cdev_add(&mdev->cdev, MKDEV(MAJOR(g_mic_devno), mdev->id), 1);
+	if (rc) {
+		dev_err(&pdev->dev, "cdev_add err id %d rc %d\n", mdev->id, rc);
+		goto cleanup_debug_dir;
+	}
 	return 0;
+cleanup_debug_dir:
+	mic_delete_debug_dir(mdev);
+	mutex_lock(&mdev->mic_mutex);
+	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);
+	mutex_unlock(&mdev->mic_mutex);
 dp_uninit:
 	mic_dp_uninit(mdev);
 sysfs_put:

@@ -996,6 +1020,7 @@ static void mic_remove(struct pci_dev *pdev)
 		return;
 
 	mic_stop(mdev, false);
+	cdev_del(&mdev->cdev);
 	mic_delete_debug_dir(mdev);
 	mutex_lock(&mdev->mic_mutex);
 	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);

drivers/misc/mic/host/mic_virtio.c new file +704 -0

704 lines not on this page · raw message

drivers/misc/mic/host/mic_virtio.h new file +138 -0

138 lines not on this page · raw message

include/uapi/linux/Kbuild +1 -0

index e3f7b37..c73dbd0 100644
@@ -241,6 +241,7 @@ header-y += mei.h
 header-y += mempolicy.h
 header-y += meye.h
 header-y += mic_common.h
+header-y += mic_ioctl.h
 header-y += mii.h
 header-y += minix_fs.h
 header-y += mman.h

include/uapi/linux/mic_common.h +165 -1

178 lines not on this page · raw message

include/uapi/linux/mic_ioctl.h new file +74 -0

74 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:52 UTC #30 lore raw

PATCH v3 7/7 Sample Implementation of Intel MIC User Space Daemon.

In reply to Sudeep Dutt

From: Caz Yokoyama <[email protected]>

Co-author: Ashutosh Dixit <[email protected]>
Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

---
 Documentation/mic/mic_overview.txt |   49 ++
 Documentation/mic/mpssd/.gitignore |    1 +
 Documentation/mic/mpssd/Makefile   |   19 +
 Documentation/mic/mpssd/micctrl    |  173 ++++
 Documentation/mic/mpssd/mpss       |  202 +++++
 Documentation/mic/mpssd/mpssd.c    | 1701 ++++++++++++++++++++++++++++++++++++
 Documentation/mic/mpssd/mpssd.h    |  100 +++
 Documentation/mic/mpssd/sysfs.c    |  102 +++
 8 files changed, 2347 insertions(+)
 create mode 100644 Documentation/mic/mic_overview.txt
 create mode 100644 Documentation/mic/mpssd/.gitignore
 create mode 100644 Documentation/mic/mpssd/Makefile
 create mode 100755 Documentation/mic/mpssd/micctrl
 create mode 100755 Documentation/mic/mpssd/mpss
 create mode 100644 Documentation/mic/mpssd/mpssd.c
 create mode 100644 Documentation/mic/mpssd/mpssd.h
 create mode 100644 Documentation/mic/mpssd/sysfs.c

Documentation/mic/mic_overview.txt new file +49 -0

new file mode 100644
index 0000000..c4424ed
@@ -0,0 +1,49 @@
+An Intel MIC X100 device is a PCIe form factor add-in coprocessor
+card based on the Intel Many Integrated Core (MIC) architecture
+that runs a Linux OS. It is a PCIe endpoint in a platform and therefore
+implements the three required standard address spaces i.e. configuration,
+memory and I/O. The host OS loads a device driver as is typical for
+PCIe devices. The card itself runs a bootstrap after reset that
+transfers control to the card OS downloaded from the host driver.
+The card OS as shipped by Intel is a Linux kernel with modifications
+for the X100 devices.
+
+Since it is a PCIe card, it does not have the ability to host hardware
+devices for networking, storage and console. We provide these devices
+on X100 coprocessors thus enabling a self-bootable equivalent environment
+for applications. A key benefit of our solution is that it leverages
+the standard virtio framework for network, disk and console devices,
+though in our case the virtio framework is used across a PCIe bus.
+
+Here is a block diagram of the various components described above. The
+virtio backends are situated on the host rather than the card given better
+single threaded performance for the host compared to MIC, the ability of
+the host to initiate DMA's to/from the card using the MIC DMA engine and
+the fact that the virtio block storage backend can only be on the host.
+
+                              |
+       +----------+           |             +----------+
+       | Card OS  |           |             | Host OS  |
+       +----------+           |             +----------+
+                              |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+| Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
+| Net   | |Console | |Block | | |Net      |  |Console | |Block   |
+| Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+    |         |         |     |      |            |         |
+    |         |         |     |User  |            |         |
+    |         |         |     |------|------------|---------|-------
+    +-------------------+     |Kernel +--------------------------+
+              |               |       | Virtio over PCIe IOCTLs  |
+              |               |       +--------------------------+
+      +--------------+        |                   |
+      |Intel MIC     |        |            +---------------+
+      |Card Driver   |        |            |Intel MIC      |
+      +--------------+        |            |Host Driver    |
+              |               |            +---------------+
+              |               |                   |
+     +-------------------------------------------------------------+
+     |                                                             |
+     |                    PCIe Bus                                 |
+     +-------------------------------------------------------------+

Documentation/mic/mpssd/.gitignore new file +1 -0

new file mode 100644
index 0000000..8b7c72f
@@ -0,0 +1 @@
+mpssd

Documentation/mic/mpssd/Makefile new file +19 -0

new file mode 100644
index 0000000..eb860a7
@@ -0,0 +1,19 @@
+#
+# Makefile - Intel MIC User Space Tools.
+# Copyright(c) 2013, Intel Corporation.
+#
+ifdef DEBUG
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall -DDEBUG=$(DEBUG)
+else
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall
+endif
+
+mpssd: mpssd.o sysfs.o
+	$(CC) $(CFLAGS) -o $@ $^ -lpthread
+
+install:
+	install mpssd /usr/sbin/mpssd
+	install micctrl /usr/sbin/micctrl
+
+clean:
+	rm -f mpssd *.o

Documentation/mic/mpssd/micctrl new file +173 -0

173 lines not on this page · raw message

Documentation/mic/mpssd/mpss new file +202 -0

202 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.c new file +1701 -0

1,701 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.h new file +100 -0

100 lines not on this page · raw message

Documentation/mic/mpssd/sysfs.c new file +102 -0

102 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:52 UTC #9 lore raw

PATCH v3 1/7 Intel MIC Host Driver for X100 family.

In reply to Sudeep Dutt

This patch enables the following:
a) Initializes the Intel MIC X100 PCIe devices.
b) Provides sysfs entries for family and stepping information.

Co-author: Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 Documentation/ABI/testing/sysfs-class-mic.txt |  34 +++
 drivers/misc/Kconfig                          |   1 +
 drivers/misc/Makefile                         |   1 +
 drivers/misc/mic/Kconfig                      |  19 ++
 drivers/misc/mic/Makefile                     |   5 +
 drivers/misc/mic/common/mic_device.h          |  37 +++
 drivers/misc/mic/host/Makefile                |   8 +
 drivers/misc/mic/host/mic_device.h            | 109 +++++++++
 drivers/misc/mic/host/mic_main.c              | 309 ++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_sysfs.c             |  97 ++++++++
 drivers/misc/mic/host/mic_x100.c              |  75 +++++++
 drivers/misc/mic/host/mic_x100.h              |  47 ++++
 12 files changed, 742 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt
 create mode 100644 drivers/misc/mic/Kconfig
 create mode 100644 drivers/misc/mic/Makefile
 create mode 100644 drivers/misc/mic/common/mic_device.h
 create mode 100644 drivers/misc/mic/host/Makefile
 create mode 100644 drivers/misc/mic/host/mic_device.h
 create mode 100644 drivers/misc/mic/host/mic_main.c
 create mode 100644 drivers/misc/mic/host/mic_sysfs.c
 create mode 100644 drivers/misc/mic/host/mic_x100.c
 create mode 100644 drivers/misc/mic/host/mic_x100.h

Documentation/ABI/testing/sysfs-class-mic.txt new file +34 -0

new file mode 100644
index 0000000..09eb3c6
@@ -0,0 +1,34 @@
+What:		/sys/class/mic/
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The mic class directory belongs to Intel MIC devices and
+		provides information per MIC device. An Intel MIC device is a
+		PCIe form factor add-in Coprocessor card based on the Intel Many
+		Integrated Core (MIC) architecture that runs a Linux OS.
+
+What:		/sys/class/mic/mic(x)
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		The directories /sys/class/mic/mic0, /sys/class/mic/mic1 etc.,
+		represent MIC devices (0,1,..etc). Each directory has
+		information specific to that MIC device.
+
+What:		/sys/class/mic/mic(x)/family
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the Coprocessor family for an Intel
+		MIC device. For example - "x100"
+
+What:		/sys/class/mic/mic(x)/stepping
+Date:		August 2013
+KernelVersion:	3.11
+Contact:	Sudeep Dutt <[email protected]>
+Description:
+		Provides information about the silicon stepping for an Intel
+		MIC device. For example - "A0" or "B0"

drivers/misc/Kconfig +1 -0

index 8dacd4c..e760715 100644
@@ -537,4 +537,5 @@ source "drivers/misc/carma/Kconfig"
 source "drivers/misc/altera-stapl/Kconfig"
 source "drivers/misc/mei/Kconfig"
 source "drivers/misc/vmw_vmci/Kconfig"
+source "drivers/misc/mic/Kconfig"
 endmenu

drivers/misc/Makefile +1 -0

index c235d5b..0b7ea3e 100644
@@ -53,3 +53,4 @@ obj-$(CONFIG_INTEL_MEI)		+= mei/
 obj-$(CONFIG_VMWARE_VMCI)	+= vmw_vmci/
 obj-$(CONFIG_LATTICE_ECP3_CONFIG)	+= lattice-ecp3-config.o
 obj-$(CONFIG_SRAM)		+= sram.o
+obj-y				+= mic/

drivers/misc/mic/Kconfig new file +19 -0

new file mode 100644
index 0000000..aaefd0c
@@ -0,0 +1,19 @@
+comment "Intel MIC Host Driver"
+
+config INTEL_MIC_HOST
+	tristate "Intel MIC Host Driver"
+	depends on 64BIT && PCI
+	default N
+	help
+	  This enables Host Driver support for the Intel Many Integrated
+	  Core (MIC) family of PCIe form factor coprocessor devices that
+	  run a 64 bit Linux OS. The driver manages card OS state and
+	  enables communication between host and card. Intel MIC X100
+	  devices are currently supported.
+
+	  If you are building a host kernel with an Intel MIC device then
+	  say M (recommended) or Y, else say N. If unsure say N.
+
+	  More information about the Intel MIC family as well as the Linux
+	  OS and tools for MIC to use with this driver are available from
+	  <http://software.intel.com/en-us/mic-developer>.

drivers/misc/mic/Makefile new file +5 -0

new file mode 100644
index 0000000..8e72421
@@ -0,0 +1,5 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += host/

drivers/misc/mic/common/mic_device.h new file +37 -0

new file mode 100644
index 0000000..f02262e
@@ -0,0 +1,37 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC driver.
+ *
+ */
+#ifndef __MIC_COMMON_DEVICE_H_
+#define __MIC_COMMON_DEVICE_H_
+
+/**
+ * struct mic_mw - MIC memory window
+ *
+ * @pa: Base physical address.
+ * @va: Base ioremap'd virtual address.
+ * @len: Size of the memory window.
+ */
+struct mic_mw {
+	phys_addr_t pa;
+	void __iomem *va;
+	resource_size_t len;
+};
+
+#endif

drivers/misc/mic/host/Makefile new file +8 -0

new file mode 100644
index 0000000..93b9d25
@@ -0,0 +1,8 @@
+#
+# Makefile - Intel MIC Linux driver.
+# Copyright(c) 2013, Intel Corporation.
+#
+obj-$(CONFIG_INTEL_MIC_HOST) += mic_host.o
+mic_host-objs := mic_main.o
+mic_host-objs += mic_x100.o
+mic_host-objs += mic_sysfs.o

drivers/misc/mic/host/mic_device.h new file +109 -0

109 lines not on this page · raw message

drivers/misc/mic/host/mic_main.c new file +309 -0

309 lines not on this page · raw message

drivers/misc/mic/host/mic_sysfs.c new file +97 -0

97 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.c new file +75 -0

75 lines not on this page · raw message

drivers/misc/mic/host/mic_x100.h new file +47 -0

47 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:52 UTC #20 lore raw

PATCH v3 5/7 Intel MIC Host Driver Changes for Virtio Devices.

In reply to Sudeep Dutt

From: Ashutosh Dixit <[email protected]>

Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 drivers/misc/mic/Kconfig             |   1 +
 drivers/misc/mic/common/mic_device.h |   7 +
 drivers/misc/mic/host/Makefile       |   2 +
 drivers/misc/mic/host/mic_boot.c     |   3 +-
 drivers/misc/mic/host/mic_debugfs.c  | 140 +++++++
 drivers/misc/mic/host/mic_device.h   |   5 +
 drivers/misc/mic/host/mic_fops.c     | 221 +++++++++++
 drivers/misc/mic/host/mic_fops.h     |  32 ++
 drivers/misc/mic/host/mic_main.c     |  27 +-
 drivers/misc/mic/host/mic_virtio.c   | 704 +++++++++++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_virtio.h   | 138 +++++++
 include/uapi/linux/Kbuild            |   1 +
 include/uapi/linux/mic_common.h      | 166 ++++++++-
 include/uapi/linux/mic_ioctl.h       |  74 ++++
 14 files changed, 1518 insertions(+), 3 deletions(-)
 create mode 100644 drivers/misc/mic/host/mic_fops.c
 create mode 100644 drivers/misc/mic/host/mic_fops.h
 create mode 100644 drivers/misc/mic/host/mic_virtio.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.h
 create mode 100644 include/uapi/linux/mic_ioctl.h

drivers/misc/mic/Kconfig +1 -0

index 279a2e6..01f1a4a 100644
@@ -3,6 +3,7 @@ comment "Intel MIC Host Driver"
 config INTEL_MIC_HOST
 	tristate "Intel MIC Host Driver"
 	depends on 64BIT && PCI
+	select VHOST_RING
 	default N
 	help
 	  This enables Host Driver support for the Intel Many Integrated

drivers/misc/mic/common/mic_device.h +7 -0

index 6440e9d..01eb74f 100644
@@ -41,4 +41,11 @@ struct mic_mw {
 #define MIC_DPLO_SPAD 14
 #define MIC_DPHI_SPAD 15
 
+/*
+ * These values are supposed to be in the config_change field of the
+ * device page when the host sends a config change interrupt to the card.
+ */
+#define MIC_VIRTIO_PARAM_DEV_REMOVE 0x1
+#define MIC_VIRTIO_PARAM_CONFIG_CHANGED 0x2
+
 #endif

drivers/misc/mic/host/Makefile +2 -0

index 98bf565..e73ac74 100644
@@ -9,3 +9,5 @@ mic_host-objs += mic_sysfs.o
 mic_host-objs += mic_smpt.o
 mic_host-objs += mic_boot.o
 mic_host-objs += mic_debugfs.o
+mic_host-objs += mic_fops.o
+mic_host-objs += mic_virtio.o

drivers/misc/mic/host/mic_boot.c +2 -1

index 936fc58..fd9ff6d 100644
@@ -20,12 +20,12 @@
  */
 #include <linux/delay.h>
 #include <linux/firmware.h>
-#include <linux/interrupt.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_smpt.h"
+#include "mic_virtio.h"
 
 /**
  * mic_reset - Reset the MIC device.

@@ -117,6 +117,7 @@ void mic_stop(struct mic_device *mdev, bool force)
 {
 	mutex_lock(&mdev->mic_mutex);
 	if (MIC_OFFLINE != mdev->state || force) {
+		mic_virtio_reset_devices(mdev);
 		mic_bootparam_init(mdev);
 		mic_reset(mdev);
 		if (MIC_RESET_FAILED == mdev->state)

drivers/misc/mic/host/mic_debugfs.c +140 -0

171 lines not on this page · raw message

drivers/misc/mic/host/mic_device.h +5 -0

index c2ca811..4592e7d 100644
@@ -21,6 +21,7 @@
 #ifndef _MIC_DEVICE_H_
 #define _MIC_DEVICE_H_
 
+#include <linux/cdev.h>
 #include <linux/idr.h>
 
 /* The maximum number of MIC devices supported in a single host system. */

@@ -156,6 +157,8 @@ struct mic_irq;
  * @dp_dma_addr: virtio device page DMA address.
  * @shutdown_db: shutdown doorbell.
  * @shutdown_cookie: shutdown cookie.
+ * @cdev: Character device for MIC.
+ * @vdev_list: list of virtio devices.
  */
 struct mic_device {
 	struct mic_mw mmio;

@@ -189,6 +192,8 @@ struct mic_device {
 	dma_addr_t dp_dma_addr;
 	int shutdown_db;
 	struct mic_irq *shutdown_cookie;
+	struct cdev cdev;
+	struct list_head vdev_list;
 };
 
 /**

drivers/misc/mic/host/mic_fops.c new file +221 -0

221 lines not on this page · raw message

drivers/misc/mic/host/mic_fops.h new file +32 -0

new file mode 100644
index 0000000..dc3893d
@@ -0,0 +1,32 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Host driver.
+ *
+ */
+#ifndef _MIC_FOPS_H_
+#define _MIC_FOPS_H_
+
+int mic_open(struct inode *inode, struct file *filp);
+int mic_release(struct inode *inode, struct file *filp);
+ssize_t mic_read(struct file *filp, char __user *buf,
+			size_t count, loff_t *pos);
+long mic_ioctl(struct file *filp, unsigned int cmd, unsigned long arg);
+int mic_mmap(struct file *f, struct vm_area_struct *vma);
+unsigned int mic_poll(struct file *f, poll_table *wait);
+
+#endif

drivers/misc/mic/host/mic_main.c +26 -1

index 2f34a26..7c4f973 100644
@@ -22,16 +22,18 @@
  * 1) Enable DMA support.
  * 2) Enable per vring interrupt support.
  */
-#include <linux/fs.h>
 #include <linux/interrupt.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/poll.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_x100.h"
 #include "mic_smpt.h"
+#include "mic_fops.h"
+#include "mic_virtio.h"
 
 static const char mic_driver_name[] = "mic";

@@ -65,6 +67,15 @@ static struct class *g_mic_class;
 /* Base device node number for MIC devices */
 static dev_t g_mic_devno;
 
+static const struct file_operations mic_fops = {
+	.open = mic_open,
+	.release = mic_release,
+	.unlocked_ioctl = mic_ioctl,
+	.poll = mic_poll,
+	.mmap = mic_mmap,
+	.owner = THIS_MODULE,
+};
+
 /* Initialize the device page */
 static int mic_dp_init(struct mic_device *mdev)
 {

@@ -814,6 +825,7 @@ mic_device_init(struct mic_device *mdev, struct pci_dev *pdev)
 	mdev->irq_info.next_avail_src = 0;
 	INIT_WORK(&mdev->reset_trigger_work, mic_reset_trigger_work);
 	INIT_WORK(&mdev->shutdown_work, mic_shutdown_work);
+	INIT_LIST_HEAD(&mdev->vdev_list);
 }
 
 /**

@@ -951,7 +963,19 @@ static int mic_probe(struct pci_dev *pdev,
 	mic_bootparam_init(mdev);
 
 	mic_create_debug_dir(mdev);
+	cdev_init(&mdev->cdev, &mic_fops);
+	mdev->cdev.owner = THIS_MODULE;
+	rc = cdev_add(&mdev->cdev, MKDEV(MAJOR(g_mic_devno), mdev->id), 1);
+	if (rc) {
+		dev_err(&pdev->dev, "cdev_add err id %d rc %d\n", mdev->id, rc);
+		goto cleanup_debug_dir;
+	}
 	return 0;
+cleanup_debug_dir:
+	mic_delete_debug_dir(mdev);
+	mutex_lock(&mdev->mic_mutex);
+	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);
+	mutex_unlock(&mdev->mic_mutex);
 dp_uninit:
 	mic_dp_uninit(mdev);
 sysfs_put:

@@ -996,6 +1020,7 @@ static void mic_remove(struct pci_dev *pdev)
 		return;
 
 	mic_stop(mdev, false);
+	cdev_del(&mdev->cdev);
 	mic_delete_debug_dir(mdev);
 	mutex_lock(&mdev->mic_mutex);
 	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);

drivers/misc/mic/host/mic_virtio.c new file +704 -0

704 lines not on this page · raw message

drivers/misc/mic/host/mic_virtio.h new file +138 -0

138 lines not on this page · raw message

include/uapi/linux/Kbuild +1 -0

index e3f7b37..c73dbd0 100644
@@ -241,6 +241,7 @@ header-y += mei.h
 header-y += mempolicy.h
 header-y += meye.h
 header-y += mic_common.h
+header-y += mic_ioctl.h
 header-y += mii.h
 header-y += minix_fs.h
 header-y += mman.h

include/uapi/linux/mic_common.h +165 -1

178 lines not on this page · raw message

include/uapi/linux/mic_ioctl.h new file +74 -0

74 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:52 UTC #31 lore raw

PATCH v3 7/7 Sample Implementation of Intel MIC User Space Daemon.

In reply to Sudeep Dutt

From: Caz Yokoyama <[email protected]>

Co-author: Ashutosh Dixit <[email protected]>
Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

---
 Documentation/mic/mic_overview.txt |   49 ++
 Documentation/mic/mpssd/.gitignore |    1 +
 Documentation/mic/mpssd/Makefile   |   19 +
 Documentation/mic/mpssd/micctrl    |  173 ++++
 Documentation/mic/mpssd/mpss       |  202 +++++
 Documentation/mic/mpssd/mpssd.c    | 1701 ++++++++++++++++++++++++++++++++++++
 Documentation/mic/mpssd/mpssd.h    |  100 +++
 Documentation/mic/mpssd/sysfs.c    |  102 +++
 8 files changed, 2347 insertions(+)
 create mode 100644 Documentation/mic/mic_overview.txt
 create mode 100644 Documentation/mic/mpssd/.gitignore
 create mode 100644 Documentation/mic/mpssd/Makefile
 create mode 100755 Documentation/mic/mpssd/micctrl
 create mode 100755 Documentation/mic/mpssd/mpss
 create mode 100644 Documentation/mic/mpssd/mpssd.c
 create mode 100644 Documentation/mic/mpssd/mpssd.h
 create mode 100644 Documentation/mic/mpssd/sysfs.c

Documentation/mic/mic_overview.txt new file +49 -0

new file mode 100644
index 0000000..c4424ed
@@ -0,0 +1,49 @@
+An Intel MIC X100 device is a PCIe form factor add-in coprocessor
+card based on the Intel Many Integrated Core (MIC) architecture
+that runs a Linux OS. It is a PCIe endpoint in a platform and therefore
+implements the three required standard address spaces i.e. configuration,
+memory and I/O. The host OS loads a device driver as is typical for
+PCIe devices. The card itself runs a bootstrap after reset that
+transfers control to the card OS downloaded from the host driver.
+The card OS as shipped by Intel is a Linux kernel with modifications
+for the X100 devices.
+
+Since it is a PCIe card, it does not have the ability to host hardware
+devices for networking, storage and console. We provide these devices
+on X100 coprocessors thus enabling a self-bootable equivalent environment
+for applications. A key benefit of our solution is that it leverages
+the standard virtio framework for network, disk and console devices,
+though in our case the virtio framework is used across a PCIe bus.
+
+Here is a block diagram of the various components described above. The
+virtio backends are situated on the host rather than the card given better
+single threaded performance for the host compared to MIC, the ability of
+the host to initiate DMA's to/from the card using the MIC DMA engine and
+the fact that the virtio block storage backend can only be on the host.
+
+                              |
+       +----------+           |             +----------+
+       | Card OS  |           |             | Host OS  |
+       +----------+           |             +----------+
+                              |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+| Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
+| Net   | |Console | |Block | | |Net      |  |Console | |Block   |
+| Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
++-------+ +--------+ +------+ | +---------+  +--------+ +--------+
+    |         |         |     |      |            |         |
+    |         |         |     |User  |            |         |
+    |         |         |     |------|------------|---------|-------
+    +-------------------+     |Kernel +--------------------------+
+              |               |       | Virtio over PCIe IOCTLs  |
+              |               |       +--------------------------+
+      +--------------+        |                   |
+      |Intel MIC     |        |            +---------------+
+      |Card Driver   |        |            |Intel MIC      |
+      +--------------+        |            |Host Driver    |
+              |               |            +---------------+
+              |               |                   |
+     +-------------------------------------------------------------+
+     |                                                             |
+     |                    PCIe Bus                                 |
+     +-------------------------------------------------------------+

Documentation/mic/mpssd/.gitignore new file +1 -0

new file mode 100644
index 0000000..8b7c72f
@@ -0,0 +1 @@
+mpssd

Documentation/mic/mpssd/Makefile new file +19 -0

new file mode 100644
index 0000000..eb860a7
@@ -0,0 +1,19 @@
+#
+# Makefile - Intel MIC User Space Tools.
+# Copyright(c) 2013, Intel Corporation.
+#
+ifdef DEBUG
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall -DDEBUG=$(DEBUG)
+else
+CFLAGS += $(USERWARNFLAGS) -I. -g -Wall
+endif
+
+mpssd: mpssd.o sysfs.o
+	$(CC) $(CFLAGS) -o $@ $^ -lpthread
+
+install:
+	install mpssd /usr/sbin/mpssd
+	install micctrl /usr/sbin/micctrl
+
+clean:
+	rm -f mpssd *.o

Documentation/mic/mpssd/micctrl new file +173 -0

173 lines not on this page · raw message

Documentation/mic/mpssd/mpss new file +202 -0

202 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.c new file +1701 -0

1,701 lines not on this page · raw message

Documentation/mic/mpssd/mpssd.h new file +100 -0

100 lines not on this page · raw message

Documentation/mic/mpssd/sysfs.c new file +102 -0

102 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:53 UTC #21 lore raw

PATCH v3 5/7 Intel MIC Host Driver Changes for Virtio Devices.

In reply to Sudeep Dutt

From: Ashutosh Dixit <[email protected]>

Co-author: Sudeep Dutt <[email protected]>

Signed-off-by Ashutosh Dixit <[email protected]>

Signed-off-by Caz Yokoyama <[email protected]>

Signed-off-by Dasaratharaman Chandramouli <[email protected]>

Signed-off-by Nikhil Rao <[email protected]>

Signed-off-by Harshavardhan R Kharche <[email protected]>

Signed-off-by Sudeep Dutt <[email protected]>

Acked-by Yaozu (Eddie) Dong <[email protected]>

Reviewed-by Peter P Waskiewicz Jr <[email protected]>

---
 drivers/misc/mic/Kconfig             |   1 +
 drivers/misc/mic/common/mic_device.h |   7 +
 drivers/misc/mic/host/Makefile       |   2 +
 drivers/misc/mic/host/mic_boot.c     |   3 +-
 drivers/misc/mic/host/mic_debugfs.c  | 140 +++++++
 drivers/misc/mic/host/mic_device.h   |   5 +
 drivers/misc/mic/host/mic_fops.c     | 221 +++++++++++
 drivers/misc/mic/host/mic_fops.h     |  32 ++
 drivers/misc/mic/host/mic_main.c     |  27 +-
 drivers/misc/mic/host/mic_virtio.c   | 704 +++++++++++++++++++++++++++++++++++
 drivers/misc/mic/host/mic_virtio.h   | 138 +++++++
 include/uapi/linux/Kbuild            |   1 +
 include/uapi/linux/mic_common.h      | 166 ++++++++-
 include/uapi/linux/mic_ioctl.h       |  74 ++++
 14 files changed, 1518 insertions(+), 3 deletions(-)
 create mode 100644 drivers/misc/mic/host/mic_fops.c
 create mode 100644 drivers/misc/mic/host/mic_fops.h
 create mode 100644 drivers/misc/mic/host/mic_virtio.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.h
 create mode 100644 include/uapi/linux/mic_ioctl.h

drivers/misc/mic/Kconfig +1 -0

index 279a2e6..01f1a4a 100644
@@ -3,6 +3,7 @@ comment "Intel MIC Host Driver"
 config INTEL_MIC_HOST
 	tristate "Intel MIC Host Driver"
 	depends on 64BIT && PCI
+	select VHOST_RING
 	default N
 	help
 	  This enables Host Driver support for the Intel Many Integrated

drivers/misc/mic/common/mic_device.h +7 -0

index 6440e9d..01eb74f 100644
@@ -41,4 +41,11 @@ struct mic_mw {
 #define MIC_DPLO_SPAD 14
 #define MIC_DPHI_SPAD 15
 
+/*
+ * These values are supposed to be in the config_change field of the
+ * device page when the host sends a config change interrupt to the card.
+ */
+#define MIC_VIRTIO_PARAM_DEV_REMOVE 0x1
+#define MIC_VIRTIO_PARAM_CONFIG_CHANGED 0x2
+
 #endif

drivers/misc/mic/host/Makefile +2 -0

index 98bf565..e73ac74 100644
@@ -9,3 +9,5 @@ mic_host-objs += mic_sysfs.o
 mic_host-objs += mic_smpt.o
 mic_host-objs += mic_boot.o
 mic_host-objs += mic_debugfs.o
+mic_host-objs += mic_fops.o
+mic_host-objs += mic_virtio.o

drivers/misc/mic/host/mic_boot.c +2 -1

index 936fc58..fd9ff6d 100644
@@ -20,12 +20,12 @@
  */
 #include <linux/delay.h>
 #include <linux/firmware.h>
-#include <linux/interrupt.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_smpt.h"
+#include "mic_virtio.h"
 
 /**
  * mic_reset - Reset the MIC device.

@@ -117,6 +117,7 @@ void mic_stop(struct mic_device *mdev, bool force)
 {
 	mutex_lock(&mdev->mic_mutex);
 	if (MIC_OFFLINE != mdev->state || force) {
+		mic_virtio_reset_devices(mdev);
 		mic_bootparam_init(mdev);
 		mic_reset(mdev);
 		if (MIC_RESET_FAILED == mdev->state)

drivers/misc/mic/host/mic_debugfs.c +140 -0

171 lines not on this page · raw message

drivers/misc/mic/host/mic_device.h +5 -0

index c2ca811..4592e7d 100644
@@ -21,6 +21,7 @@
 #ifndef _MIC_DEVICE_H_
 #define _MIC_DEVICE_H_
 
+#include <linux/cdev.h>
 #include <linux/idr.h>
 
 /* The maximum number of MIC devices supported in a single host system. */

@@ -156,6 +157,8 @@ struct mic_irq;
  * @dp_dma_addr: virtio device page DMA address.
  * @shutdown_db: shutdown doorbell.
  * @shutdown_cookie: shutdown cookie.
+ * @cdev: Character device for MIC.
+ * @vdev_list: list of virtio devices.
  */
 struct mic_device {
 	struct mic_mw mmio;

@@ -189,6 +192,8 @@ struct mic_device {
 	dma_addr_t dp_dma_addr;
 	int shutdown_db;
 	struct mic_irq *shutdown_cookie;
+	struct cdev cdev;
+	struct list_head vdev_list;
 };
 
 /**

drivers/misc/mic/host/mic_fops.c new file +221 -0

221 lines not on this page · raw message

drivers/misc/mic/host/mic_fops.h new file +32 -0

new file mode 100644
index 0000000..dc3893d
@@ -0,0 +1,32 @@
+/*
+ * Intel MIC Platform Software Stack (MPSS)
+ *
+ * Copyright(c) 2013 Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * The full GNU General Public License is included in this distribution in
+ * the file called "COPYING".
+ *
+ * Intel MIC Host driver.
+ *
+ */
+#ifndef _MIC_FOPS_H_
+#define _MIC_FOPS_H_
+
+int mic_open(struct inode *inode, struct file *filp);
+int mic_release(struct inode *inode, struct file *filp);
+ssize_t mic_read(struct file *filp, char __user *buf,
+			size_t count, loff_t *pos);
+long mic_ioctl(struct file *filp, unsigned int cmd, unsigned long arg);
+int mic_mmap(struct file *f, struct vm_area_struct *vma);
+unsigned int mic_poll(struct file *f, poll_table *wait);
+
+#endif

drivers/misc/mic/host/mic_main.c +26 -1

index 2f34a26..7c4f973 100644
@@ -22,16 +22,18 @@
  * 1) Enable DMA support.
  * 2) Enable per vring interrupt support.
  */
-#include <linux/fs.h>
 #include <linux/interrupt.h>
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <linux/poll.h>
 
 #include <linux/mic_common.h>
 #include "../common/mic_device.h"
 #include "mic_device.h"
 #include "mic_x100.h"
 #include "mic_smpt.h"
+#include "mic_fops.h"
+#include "mic_virtio.h"
 
 static const char mic_driver_name[] = "mic";

@@ -65,6 +67,15 @@ static struct class *g_mic_class;
 /* Base device node number for MIC devices */
 static dev_t g_mic_devno;
 
+static const struct file_operations mic_fops = {
+	.open = mic_open,
+	.release = mic_release,
+	.unlocked_ioctl = mic_ioctl,
+	.poll = mic_poll,
+	.mmap = mic_mmap,
+	.owner = THIS_MODULE,
+};
+
 /* Initialize the device page */
 static int mic_dp_init(struct mic_device *mdev)
 {

@@ -814,6 +825,7 @@ mic_device_init(struct mic_device *mdev, struct pci_dev *pdev)
 	mdev->irq_info.next_avail_src = 0;
 	INIT_WORK(&mdev->reset_trigger_work, mic_reset_trigger_work);
 	INIT_WORK(&mdev->shutdown_work, mic_shutdown_work);
+	INIT_LIST_HEAD(&mdev->vdev_list);
 }
 
 /**

@@ -951,7 +963,19 @@ static int mic_probe(struct pci_dev *pdev,
 	mic_bootparam_init(mdev);
 
 	mic_create_debug_dir(mdev);
+	cdev_init(&mdev->cdev, &mic_fops);
+	mdev->cdev.owner = THIS_MODULE;
+	rc = cdev_add(&mdev->cdev, MKDEV(MAJOR(g_mic_devno), mdev->id), 1);
+	if (rc) {
+		dev_err(&pdev->dev, "cdev_add err id %d rc %d\n", mdev->id, rc);
+		goto cleanup_debug_dir;
+	}
 	return 0;
+cleanup_debug_dir:
+	mic_delete_debug_dir(mdev);
+	mutex_lock(&mdev->mic_mutex);
+	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);
+	mutex_unlock(&mdev->mic_mutex);
 dp_uninit:
 	mic_dp_uninit(mdev);
 sysfs_put:

@@ -996,6 +1020,7 @@ static void mic_remove(struct pci_dev *pdev)
 		return;
 
 	mic_stop(mdev, false);
+	cdev_del(&mdev->cdev);
 	mic_delete_debug_dir(mdev);
 	mutex_lock(&mdev->mic_mutex);
 	mic_free_irq(mdev, mdev->shutdown_cookie, mdev);

drivers/misc/mic/host/mic_virtio.c new file +704 -0

704 lines not on this page · raw message

drivers/misc/mic/host/mic_virtio.h new file +138 -0

138 lines not on this page · raw message

include/uapi/linux/Kbuild +1 -0

index e3f7b37..c73dbd0 100644
@@ -241,6 +241,7 @@ header-y += mei.h
 header-y += mempolicy.h
 header-y += meye.h
 header-y += mic_common.h
+header-y += mic_ioctl.h
 header-y += mii.h
 header-y += minix_fs.h
 header-y += mman.h

include/uapi/linux/mic_common.h +165 -1

178 lines not on this page · raw message

include/uapi/linux/mic_ioctl.h new file +74 -0

74 lines not on this page · raw message

Signature · 1 line

1.8.2.1

Sudeep Dutt

author

2013-08-21 17:55 UTC #32 lore raw

In reply to Sudeep Dutt

Sincere apologies for the spam. This patches series got sent out 7 times by mistake due to script bug at my end. I will ensure this never happens again.

Apologies again.
Sudeep Dutt

Quoted 235 lines · Sudeep Dutt

ChangeLog:
=========
v2 => v3:
a) Patch 1 data structure cleanups, header file include cleanups, IDA interface reuse and switching to device_create_with_groups(..) as per feedback from Greg Kroah-Hartman.
b) Patch 7 signal documentation, sleep workaround removal and sysfs access API cleanups as per feedback from Michael S. Tsirkin.
v1 => v2: @ http://lwn.net/Articles/563131/
a) License wording cleanup, sysfs ABI documentation, patch 1 refactoring into 3 smaller patches and function renames, as per feedback from Greg Kroah-Hartman.
b) Use VRINGH infrastructure for accessing virtio rings from the host in patch 5, as per feedback from Michael S. Tsirkin.
v1: Initial post @ https://lwn.net/Articles/561314/
Description:
============
An Intel MIC X100 device is a PCIe form factor add-in coprocessor card based on the Intel Many Integrated Core (MIC) architecture that runs a Linux OS. It is a PCIe endpoint in a platform and therefore implements the three required standard address spaces i.e. configuration, memory and I/O. The host OS loads a device driver as is typical for PCIe devices. The card itself runs a bootstrap after reset that transfers control to the card OS downloaded from the host driver. The card OS as shipped by Intel is a Linux kernel with modifications for the X100 devices.
Since it is a PCIe card, it does not have the ability to host hardware devices for networking, storage and console. We provide these devices on X100 coprocessors thus enabling a self-bootable equivalent environment for applications. A key benefit of our solution is that it leverages the standard virtio framework for network, disk and console devices, though in our case the virtio framework is used across a PCIe bus.
Here is a block diagram of the various components described above. The virtio backends are situated on the host rather than the card given better single threaded performance for the host compared to MIC, the ability of the host to initiate DMA's to/from the card using the MIC DMA engine and the fact that the virtio block storage backend can only be on the host.
       +----------+           |             +----------+
       | Card OS  |           |             | Host OS  |
       +----------+           |             +----------+
                              |
+-------+ +--------+ +------+ | +---------+  +--------+ +--------+
| Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
| Net   | |Console | |Block | | |Net      |  |Console | |Block   |
| Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
+-------+ +--------+ +------+ | +---------+  +--------+ +--------+
    |         |         |     |      |            |         |
    |         |         |     |User  |            |         |
    |         |         |     |------|------------|---------|-------
    +-------------------+     |Kernel +--------------------------+
              |               |       | Virtio over PCIe IOCTLs  |
              |               |       +--------------------------+
      +--------------+        |                   |
      |Intel MIC     |        |            +---------------+
      |Card Driver   |        |            |Intel MIC      |
      +--------------+        |            |Host Driver    |
              |               |            +---------------+
              |               |                   |
     +-------------------------------------------------------------+
     |                                                             |
     |                    PCIe Bus                                 |
     +-------------------------------------------------------------+
The following series of patches are partitioned as follows:
Patch 1: This patch introduces the "Intel MIC Host Driver" in the block diagram which does the following:
a) Initializes the Intel MIC X100 PCIe devices.
b) Provides sysfs entries for family and stepping information.
Patch 2: This patch enables the following features in the "Intel MIC Host Driver" in the block diagram:
a) MSIx, MSI and legacy interrupt support.
b) System Memory Page Table(SMPT) support. SMPT enables system memory access from the card. On X100 devices the host can program 32 SMPT registers each capable of accessing 16GB of system memory address space from X100 devices. The registers can thereby be used to access a cumulative 512GB of system memory address space from X100 devices at any point in time.
Patch 3: This patch enables the following features in the "Intel MIC Host Driver" in the block diagram:
a) Boots and shuts down the card via sysfs entries.
b) Allocates and maps a device page for communication with the card driver and updates the device page address via scratchpad registers.
c) Provides sysfs entries for shutdown status, kernel command line, firmware, ramdisk, bootmode and log buffer information.
Patch 4: This patch introduces the "Intel MIC Card Driver" in the block diagram which does the following:
a) Initializes the Intel MIC X100 platform device and driver.
b) Sets up support to handle shutdown requests from the host.
c) Maps the device page after obtaining the device page address from the scratchpad registers updated by the host.
d) Informs the host upon a card crash by registering a panic notifier.
e) Informs the host upon a poweroff/halt event.
Patch 5: This patch introduces the host "Virtio over PCIe" interface for Intel MIC. It allows creating user space backends on the host and instantiating virtio devices for them on the Intel MIC card. It uses the existing VRINGH infrastructure in the kernel to access virtio rings from the host. A character device per MIC is exposed with IOCTL, mmap and poll callbacks. This allows the user space backend to:
(a) add/remove a virtio device via a device page.
(b) map (R/O) virtio rings and device page to user space.
(c) poll for availability of data.
(d) copy a descriptor or entire descriptor chain to/from the card.
(e) modify virtio configuration.
(f) handle virtio device reset.
The buffers are copied over using CPU copies for this initial patch and host initiated MIC DMA support is planned for future patches. The avail and desc virtio rings are in host memory and the used ring is in card memory to maximize writes across PCIe for performance.
Patch 6: This patch introduces the card "Virtio over PCIe" interface for Intel MIC. It allows virtio drivers on the card to communicate with their user space backends on the host via a device page. Ring 3 apps on the host can add, remove and configure virtio devices. A thin MIC specific virtio_config_ops is implemented which is borrowed heavily from previous similar implementations in lguest and s390 @ drivers/lguest/lguest_device.c
drivers/s390/kvm/kvm_virtio.c
Patch 7: This patch introduces a sample user space daemon which implements the virtio device backends on the host. The daemon creates/removes/configures virtio device backends by communicating with the Intel MIC Host Driver. The virtio devices currently supported are virtio net, virtio console and virtio block. Virtio net supports TSO/GSO. The daemon also monitors card shutdown status and takes appropriate actions like killing the virtio backends and resetting the card upon card shutdown and crashes.
The patches have been compiled/validated against v3.11-rc5.
Ashutosh Dixit (2):
Intel MIC Host Driver Changes for Virtio Devices. Intel MIC Card Driver Changes for Virtio Devices.
Caz Yokoyama (1):
Sample Implementation of Intel MIC User Space Daemon.
Dasaratharaman Chandramouli (1):
Intel MIC Host Driver Interrupt/SMPT support.
Sudeep Dutt (3):
Intel MIC Host Driver for X100 family. Intel MIC Host Driver, card OS state management. Intel MIC Card Driver for X100 family.
 Documentation/ABI/testing/sysfs-class-mic.txt |  147 +++
 Documentation/mic/mic_overview.txt            |   49 +
 Documentation/mic/mpssd/.gitignore            |    1 +
 Documentation/mic/mpssd/Makefile              |   19 +
 Documentation/mic/mpssd/micctrl               |  173 +++
 Documentation/mic/mpssd/mpss                  |  202 +++
 Documentation/mic/mpssd/mpssd.c               | 1701 +++++++++++++++++++++++++
 Documentation/mic/mpssd/mpssd.h               |  100 ++
 Documentation/mic/mpssd/sysfs.c               |  102 ++
 drivers/misc/Kconfig                          |    1 +
 drivers/misc/Makefile                         |    1 +
 drivers/misc/mic/Kconfig                      |   39 +
 drivers/misc/mic/Makefile                     |    6 +
 drivers/misc/mic/card/Makefile                |   11 +
 drivers/misc/mic/card/mic_debugfs.c           |  130 ++
 drivers/misc/mic/card/mic_device.c            |  306 +++++
 drivers/misc/mic/card/mic_device.h            |  133 ++
 drivers/misc/mic/card/mic_virtio.c            |  631 +++++++++
 drivers/misc/mic/card/mic_virtio.h            |   77 ++
 drivers/misc/mic/card/mic_x100.c              |  256 ++++
 drivers/misc/mic/card/mic_x100.h              |   48 +
 drivers/misc/mic/common/mic_device.h          |   51 +
 drivers/misc/mic/host/Makefile                |   13 +
 drivers/misc/mic/host/mic_boot.c              |  185 +++
 drivers/misc/mic/host/mic_debugfs.c           |  496 +++++++
 drivers/misc/mic/host/mic_device.h            |  300 +++++
 drivers/misc/mic/host/mic_fops.c              |  221 ++++
 drivers/misc/mic/host/mic_fops.h              |   32 +
 drivers/misc/mic/host/mic_main.c              | 1098 ++++++++++++++++
 drivers/misc/mic/host/mic_smpt.c              |  440 +++++++
 drivers/misc/mic/host/mic_smpt.h              |   98 ++
 drivers/misc/mic/host/mic_sysfs.c             |  468 +++++++
 drivers/misc/mic/host/mic_virtio.c            |  704 ++++++++++
 drivers/misc/mic/host/mic_virtio.h            |  138 ++
 drivers/misc/mic/host/mic_x100.c              |  573 +++++++++
 drivers/misc/mic/host/mic_x100.h              |   99 ++
 include/uapi/linux/Kbuild                     |    2 +
 include/uapi/linux/mic_common.h               |  238 ++++
 include/uapi/linux/mic_ioctl.h                |   74 ++
 39 files changed, 9363 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt
 create mode 100644 Documentation/mic/mic_overview.txt
 create mode 100644 Documentation/mic/mpssd/.gitignore
 create mode 100644 Documentation/mic/mpssd/Makefile
 create mode 100755 Documentation/mic/mpssd/micctrl
 create mode 100755 Documentation/mic/mpssd/mpss
 create mode 100644 Documentation/mic/mpssd/mpssd.c
 create mode 100644 Documentation/mic/mpssd/mpssd.h
 create mode 100644 Documentation/mic/mpssd/sysfs.c
 create mode 100644 drivers/misc/mic/Kconfig
 create mode 100644 drivers/misc/mic/Makefile
 create mode 100644 drivers/misc/mic/card/Makefile
 create mode 100644 drivers/misc/mic/card/mic_debugfs.c
 create mode 100644 drivers/misc/mic/card/mic_device.c
 create mode 100644 drivers/misc/mic/card/mic_device.h
 create mode 100644 drivers/misc/mic/card/mic_virtio.c
 create mode 100644 drivers/misc/mic/card/mic_virtio.h
 create mode 100644 drivers/misc/mic/card/mic_x100.c
 create mode 100644 drivers/misc/mic/card/mic_x100.h
 create mode 100644 drivers/misc/mic/common/mic_device.h
 create mode 100644 drivers/misc/mic/host/Makefile
 create mode 100644 drivers/misc/mic/host/mic_boot.c
 create mode 100644 drivers/misc/mic/host/mic_debugfs.c
 create mode 100644 drivers/misc/mic/host/mic_device.h
 create mode 100644 drivers/misc/mic/host/mic_fops.c
 create mode 100644 drivers/misc/mic/host/mic_fops.h
 create mode 100644 drivers/misc/mic/host/mic_main.c
 create mode 100644 drivers/misc/mic/host/mic_smpt.c
 create mode 100644 drivers/misc/mic/host/mic_smpt.h
 create mode 100644 drivers/misc/mic/host/mic_sysfs.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.h
 create mode 100644 drivers/misc/mic/host/mic_x100.c
 create mode 100644 drivers/misc/mic/host/mic_x100.h
 create mode 100644 include/uapi/linux/mic_common.h
 create mode 100644 include/uapi/linux/mic_ioctl.h

Sudeep Dutt

author

2013-08-21 17:56 UTC #33 lore raw

In reply to Sudeep Dutt

Sincere apologies for the spam. This patches series got sent out 7 times by mistake due to script bug at my end. I will ensure this never happens again.

Apologies again.
Sudeep Dutt

Quoted 235 lines · Sudeep Dutt

ChangeLog:
=========
v2 => v3:
a) Patch 1 data structure cleanups, header file include cleanups, IDA interface reuse and switching to device_create_with_groups(..) as per feedback from Greg Kroah-Hartman.
b) Patch 7 signal documentation, sleep workaround removal and sysfs access API cleanups as per feedback from Michael S. Tsirkin.
v1 => v2: @ http://lwn.net/Articles/563131/
a) License wording cleanup, sysfs ABI documentation, patch 1 refactoring into 3 smaller patches and function renames, as per feedback from Greg Kroah-Hartman.
b) Use VRINGH infrastructure for accessing virtio rings from the host in patch 5, as per feedback from Michael S. Tsirkin.
v1: Initial post @ https://lwn.net/Articles/561314/
Description:
============
An Intel MIC X100 device is a PCIe form factor add-in coprocessor card based on the Intel Many Integrated Core (MIC) architecture that runs a Linux OS. It is a PCIe endpoint in a platform and therefore implements the three required standard address spaces i.e. configuration, memory and I/O. The host OS loads a device driver as is typical for PCIe devices. The card itself runs a bootstrap after reset that transfers control to the card OS downloaded from the host driver. The card OS as shipped by Intel is a Linux kernel with modifications for the X100 devices.
Since it is a PCIe card, it does not have the ability to host hardware devices for networking, storage and console. We provide these devices on X100 coprocessors thus enabling a self-bootable equivalent environment for applications. A key benefit of our solution is that it leverages the standard virtio framework for network, disk and console devices, though in our case the virtio framework is used across a PCIe bus.
Here is a block diagram of the various components described above. The virtio backends are situated on the host rather than the card given better single threaded performance for the host compared to MIC, the ability of the host to initiate DMA's to/from the card using the MIC DMA engine and the fact that the virtio block storage backend can only be on the host.
       +----------+           |             +----------+
       | Card OS  |           |             | Host OS  |
       +----------+           |             +----------+
                              |
+-------+ +--------+ +------+ | +---------+  +--------+ +--------+
| Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
| Net   | |Console | |Block | | |Net      |  |Console | |Block   |
| Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
+-------+ +--------+ +------+ | +---------+  +--------+ +--------+
    |         |         |     |      |            |         |
    |         |         |     |User  |            |         |
    |         |         |     |------|------------|---------|-------
    +-------------------+     |Kernel +--------------------------+
              |               |       | Virtio over PCIe IOCTLs  |
              |               |       +--------------------------+
      +--------------+        |                   |
      |Intel MIC     |        |            +---------------+
      |Card Driver   |        |            |Intel MIC      |
      +--------------+        |            |Host Driver    |
              |               |            +---------------+
              |               |                   |
     +-------------------------------------------------------------+
     |                                                             |
     |                    PCIe Bus                                 |
     +-------------------------------------------------------------+
The following series of patches are partitioned as follows:
Patch 1: This patch introduces the "Intel MIC Host Driver" in the block diagram which does the following:
a) Initializes the Intel MIC X100 PCIe devices.
b) Provides sysfs entries for family and stepping information.
Patch 2: This patch enables the following features in the "Intel MIC Host Driver" in the block diagram:
a) MSIx, MSI and legacy interrupt support.
b) System Memory Page Table(SMPT) support. SMPT enables system memory access from the card. On X100 devices the host can program 32 SMPT registers each capable of accessing 16GB of system memory address space from X100 devices. The registers can thereby be used to access a cumulative 512GB of system memory address space from X100 devices at any point in time.
Patch 3: This patch enables the following features in the "Intel MIC Host Driver" in the block diagram:
a) Boots and shuts down the card via sysfs entries.
b) Allocates and maps a device page for communication with the card driver and updates the device page address via scratchpad registers.
c) Provides sysfs entries for shutdown status, kernel command line, firmware, ramdisk, bootmode and log buffer information.
Patch 4: This patch introduces the "Intel MIC Card Driver" in the block diagram which does the following:
a) Initializes the Intel MIC X100 platform device and driver.
b) Sets up support to handle shutdown requests from the host.
c) Maps the device page after obtaining the device page address from the scratchpad registers updated by the host.
d) Informs the host upon a card crash by registering a panic notifier.
e) Informs the host upon a poweroff/halt event.
Patch 5: This patch introduces the host "Virtio over PCIe" interface for Intel MIC. It allows creating user space backends on the host and instantiating virtio devices for them on the Intel MIC card. It uses the existing VRINGH infrastructure in the kernel to access virtio rings from the host. A character device per MIC is exposed with IOCTL, mmap and poll callbacks. This allows the user space backend to:
(a) add/remove a virtio device via a device page.
(b) map (R/O) virtio rings and device page to user space.
(c) poll for availability of data.
(d) copy a descriptor or entire descriptor chain to/from the card.
(e) modify virtio configuration.
(f) handle virtio device reset.
The buffers are copied over using CPU copies for this initial patch and host initiated MIC DMA support is planned for future patches. The avail and desc virtio rings are in host memory and the used ring is in card memory to maximize writes across PCIe for performance.
Patch 6: This patch introduces the card "Virtio over PCIe" interface for Intel MIC. It allows virtio drivers on the card to communicate with their user space backends on the host via a device page. Ring 3 apps on the host can add, remove and configure virtio devices. A thin MIC specific virtio_config_ops is implemented which is borrowed heavily from previous similar implementations in lguest and s390 @ drivers/lguest/lguest_device.c
drivers/s390/kvm/kvm_virtio.c
Patch 7: This patch introduces a sample user space daemon which implements the virtio device backends on the host. The daemon creates/removes/configures virtio device backends by communicating with the Intel MIC Host Driver. The virtio devices currently supported are virtio net, virtio console and virtio block. Virtio net supports TSO/GSO. The daemon also monitors card shutdown status and takes appropriate actions like killing the virtio backends and resetting the card upon card shutdown and crashes.
The patches have been compiled/validated against v3.11-rc5.
Ashutosh Dixit (2):
Intel MIC Host Driver Changes for Virtio Devices. Intel MIC Card Driver Changes for Virtio Devices.
Caz Yokoyama (1):
Sample Implementation of Intel MIC User Space Daemon.
Dasaratharaman Chandramouli (1):
Intel MIC Host Driver Interrupt/SMPT support.
Sudeep Dutt (3):
Intel MIC Host Driver for X100 family. Intel MIC Host Driver, card OS state management. Intel MIC Card Driver for X100 family.
 Documentation/ABI/testing/sysfs-class-mic.txt |  147 +++
 Documentation/mic/mic_overview.txt            |   49 +
 Documentation/mic/mpssd/.gitignore            |    1 +
 Documentation/mic/mpssd/Makefile              |   19 +
 Documentation/mic/mpssd/micctrl               |  173 +++
 Documentation/mic/mpssd/mpss                  |  202 +++
 Documentation/mic/mpssd/mpssd.c               | 1701 +++++++++++++++++++++++++
 Documentation/mic/mpssd/mpssd.h               |  100 ++
 Documentation/mic/mpssd/sysfs.c               |  102 ++
 drivers/misc/Kconfig                          |    1 +
 drivers/misc/Makefile                         |    1 +
 drivers/misc/mic/Kconfig                      |   39 +
 drivers/misc/mic/Makefile                     |    6 +
 drivers/misc/mic/card/Makefile                |   11 +
 drivers/misc/mic/card/mic_debugfs.c           |  130 ++
 drivers/misc/mic/card/mic_device.c            |  306 +++++
 drivers/misc/mic/card/mic_device.h            |  133 ++
 drivers/misc/mic/card/mic_virtio.c            |  631 +++++++++
 drivers/misc/mic/card/mic_virtio.h            |   77 ++
 drivers/misc/mic/card/mic_x100.c              |  256 ++++
 drivers/misc/mic/card/mic_x100.h              |   48 +
 drivers/misc/mic/common/mic_device.h          |   51 +
 drivers/misc/mic/host/Makefile                |   13 +
 drivers/misc/mic/host/mic_boot.c              |  185 +++
 drivers/misc/mic/host/mic_debugfs.c           |  496 +++++++
 drivers/misc/mic/host/mic_device.h            |  300 +++++
 drivers/misc/mic/host/mic_fops.c              |  221 ++++
 drivers/misc/mic/host/mic_fops.h              |   32 +
 drivers/misc/mic/host/mic_main.c              | 1098 ++++++++++++++++
 drivers/misc/mic/host/mic_smpt.c              |  440 +++++++
 drivers/misc/mic/host/mic_smpt.h              |   98 ++
 drivers/misc/mic/host/mic_sysfs.c             |  468 +++++++
 drivers/misc/mic/host/mic_virtio.c            |  704 ++++++++++
 drivers/misc/mic/host/mic_virtio.h            |  138 ++
 drivers/misc/mic/host/mic_x100.c              |  573 +++++++++
 drivers/misc/mic/host/mic_x100.h              |   99 ++
 include/uapi/linux/Kbuild                     |    2 +
 include/uapi/linux/mic_common.h               |  238 ++++
 include/uapi/linux/mic_ioctl.h                |   74 ++
 39 files changed, 9363 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt
 create mode 100644 Documentation/mic/mic_overview.txt
 create mode 100644 Documentation/mic/mpssd/.gitignore
 create mode 100644 Documentation/mic/mpssd/Makefile
 create mode 100755 Documentation/mic/mpssd/micctrl
 create mode 100755 Documentation/mic/mpssd/mpss
 create mode 100644 Documentation/mic/mpssd/mpssd.c
 create mode 100644 Documentation/mic/mpssd/mpssd.h
 create mode 100644 Documentation/mic/mpssd/sysfs.c
 create mode 100644 drivers/misc/mic/Kconfig
 create mode 100644 drivers/misc/mic/Makefile
 create mode 100644 drivers/misc/mic/card/Makefile
 create mode 100644 drivers/misc/mic/card/mic_debugfs.c
 create mode 100644 drivers/misc/mic/card/mic_device.c
 create mode 100644 drivers/misc/mic/card/mic_device.h
 create mode 100644 drivers/misc/mic/card/mic_virtio.c
 create mode 100644 drivers/misc/mic/card/mic_virtio.h
 create mode 100644 drivers/misc/mic/card/mic_x100.c
 create mode 100644 drivers/misc/mic/card/mic_x100.h
 create mode 100644 drivers/misc/mic/common/mic_device.h
 create mode 100644 drivers/misc/mic/host/Makefile
 create mode 100644 drivers/misc/mic/host/mic_boot.c
 create mode 100644 drivers/misc/mic/host/mic_debugfs.c
 create mode 100644 drivers/misc/mic/host/mic_device.h
 create mode 100644 drivers/misc/mic/host/mic_fops.c
 create mode 100644 drivers/misc/mic/host/mic_fops.h
 create mode 100644 drivers/misc/mic/host/mic_main.c
 create mode 100644 drivers/misc/mic/host/mic_smpt.c
 create mode 100644 drivers/misc/mic/host/mic_smpt.h
 create mode 100644 drivers/misc/mic/host/mic_sysfs.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.h
 create mode 100644 drivers/misc/mic/host/mic_x100.c
 create mode 100644 drivers/misc/mic/host/mic_x100.h
 create mode 100644 include/uapi/linux/mic_common.h
 create mode 100644 include/uapi/linux/mic_ioctl.h

Sudeep Dutt

author

2013-09-04 16:18 UTC #34 lore raw

In reply to Sudeep Dutt

Quoted 10 lines · Sudeep Dutt

ChangeLog:
=========
v2 => v3:
a) Patch 1 data structure cleanups, header file include cleanups, IDA interface reuse and switching to device_create_with_groups(..) as per feedback from Greg Kroah-Hartman.
b) Patch 7 signal documentation, sleep workaround removal and sysfs access API cleanups as per feedback from Michael S. Tsirkin.

Quoted 9 lines

v1 => v2: @ http://lwn.net/Articles/563131/
a) License wording cleanup, sysfs ABI documentation, patch 1 refactoring into 3 smaller patches and function renames, as per feedback from Greg Kroah-Hartman.
b) Use VRINGH infrastructure for accessing virtio rings from the host in patch 5, as per feedback from Michael S. Tsirkin.
v1: Initial post @ https://lwn.net/Articles/561314/

Hi Greg,

It has been a couple of weeks since we posted v3 @
https://lwn.net/Articles/564795 which addresses all feedback received so far. Have you had a chance to review this version? Please let us know your thoughts.

Thanks in advance,
Sudeep Dutt

Quoted 216 lines

Description:
============
An Intel MIC X100 device is a PCIe form factor add-in coprocessor card based on the Intel Many Integrated Core (MIC) architecture that runs a Linux OS. It is a PCIe endpoint in a platform and therefore implements the three required standard address spaces i.e. configuration, memory and I/O. The host OS loads a device driver as is typical for PCIe devices. The card itself runs a bootstrap after reset that transfers control to the card OS downloaded from the host driver. The card OS as shipped by Intel is a Linux kernel with modifications for the X100 devices.
Since it is a PCIe card, it does not have the ability to host hardware devices for networking, storage and console. We provide these devices on X100 coprocessors thus enabling a self-bootable equivalent environment for applications. A key benefit of our solution is that it leverages the standard virtio framework for network, disk and console devices, though in our case the virtio framework is used across a PCIe bus.
Here is a block diagram of the various components described above. The virtio backends are situated on the host rather than the card given better single threaded performance for the host compared to MIC, the ability of the host to initiate DMA's to/from the card using the MIC DMA engine and the fact that the virtio block storage backend can only be on the host.
       +----------+           |             +----------+
       | Card OS  |           |             | Host OS  |
       +----------+           |             +----------+
                              |
+-------+ +--------+ +------+ | +---------+  +--------+ +--------+
| Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
| Net   | |Console | |Block | | |Net      |  |Console | |Block   |
| Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
+-------+ +--------+ +------+ | +---------+  +--------+ +--------+
    |         |         |     |      |            |         |
    |         |         |     |User  |            |         |
    |         |         |     |------|------------|---------|-------
    +-------------------+     |Kernel +--------------------------+
              |               |       | Virtio over PCIe IOCTLs  |
              |               |       +--------------------------+
      +--------------+        |                   |
      |Intel MIC     |        |            +---------------+
      |Card Driver   |        |            |Intel MIC      |
      +--------------+        |            |Host Driver    |
              |               |            +---------------+
              |               |                   |
     +-------------------------------------------------------------+
     |                                                             |
     |                    PCIe Bus                                 |
     +-------------------------------------------------------------+
The following series of patches are partitioned as follows:
Patch 1: This patch introduces the "Intel MIC Host Driver" in the block diagram which does the following:
a) Initializes the Intel MIC X100 PCIe devices.
b) Provides sysfs entries for family and stepping information.
Patch 2: This patch enables the following features in the "Intel MIC Host Driver" in the block diagram:
a) MSIx, MSI and legacy interrupt support.
b) System Memory Page Table(SMPT) support. SMPT enables system memory access from the card. On X100 devices the host can program 32 SMPT registers each capable of accessing 16GB of system memory address space from X100 devices. The registers can thereby be used to access a cumulative 512GB of system memory address space from X100 devices at any point in time.
Patch 3: This patch enables the following features in the "Intel MIC Host Driver" in the block diagram:
a) Boots and shuts down the card via sysfs entries.
b) Allocates and maps a device page for communication with the card driver and updates the device page address via scratchpad registers.
c) Provides sysfs entries for shutdown status, kernel command line, firmware, ramdisk, bootmode and log buffer information.
Patch 4: This patch introduces the "Intel MIC Card Driver" in the block diagram which does the following:
a) Initializes the Intel MIC X100 platform device and driver.
b) Sets up support to handle shutdown requests from the host.
c) Maps the device page after obtaining the device page address from the scratchpad registers updated by the host.
d) Informs the host upon a card crash by registering a panic notifier.
e) Informs the host upon a poweroff/halt event.
Patch 5: This patch introduces the host "Virtio over PCIe" interface for Intel MIC. It allows creating user space backends on the host and instantiating virtio devices for them on the Intel MIC card. It uses the existing VRINGH infrastructure in the kernel to access virtio rings from the host. A character device per MIC is exposed with IOCTL, mmap and poll callbacks. This allows the user space backend to:
(a) add/remove a virtio device via a device page.
(b) map (R/O) virtio rings and device page to user space.
(c) poll for availability of data.
(d) copy a descriptor or entire descriptor chain to/from the card.
(e) modify virtio configuration.
(f) handle virtio device reset.
The buffers are copied over using CPU copies for this initial patch and host initiated MIC DMA support is planned for future patches. The avail and desc virtio rings are in host memory and the used ring is in card memory to maximize writes across PCIe for performance.
Patch 6: This patch introduces the card "Virtio over PCIe" interface for Intel MIC. It allows virtio drivers on the card to communicate with their user space backends on the host via a device page. Ring 3 apps on the host can add, remove and configure virtio devices. A thin MIC specific virtio_config_ops is implemented which is borrowed heavily from previous similar implementations in lguest and s390 @ drivers/lguest/lguest_device.c
drivers/s390/kvm/kvm_virtio.c
Patch 7: This patch introduces a sample user space daemon which implements the virtio device backends on the host. The daemon creates/removes/configures virtio device backends by communicating with the Intel MIC Host Driver. The virtio devices currently supported are virtio net, virtio console and virtio block. Virtio net supports TSO/GSO. The daemon also monitors card shutdown status and takes appropriate actions like killing the virtio backends and resetting the card upon card shutdown and crashes.
The patches have been compiled/validated against v3.11-rc5.
Ashutosh Dixit (2):
Intel MIC Host Driver Changes for Virtio Devices. Intel MIC Card Driver Changes for Virtio Devices.
Caz Yokoyama (1):
Sample Implementation of Intel MIC User Space Daemon.
Dasaratharaman Chandramouli (1):
Intel MIC Host Driver Interrupt/SMPT support.
Sudeep Dutt (3):
Intel MIC Host Driver for X100 family. Intel MIC Host Driver, card OS state management. Intel MIC Card Driver for X100 family.
 Documentation/ABI/testing/sysfs-class-mic.txt |  147 +++
 Documentation/mic/mic_overview.txt            |   49 +
 Documentation/mic/mpssd/.gitignore            |    1 +
 Documentation/mic/mpssd/Makefile              |   19 +
 Documentation/mic/mpssd/micctrl               |  173 +++
 Documentation/mic/mpssd/mpss                  |  202 +++
 Documentation/mic/mpssd/mpssd.c               | 1701 +++++++++++++++++++++++++
 Documentation/mic/mpssd/mpssd.h               |  100 ++
 Documentation/mic/mpssd/sysfs.c               |  102 ++
 drivers/misc/Kconfig                          |    1 +
 drivers/misc/Makefile                         |    1 +
 drivers/misc/mic/Kconfig                      |   39 +
 drivers/misc/mic/Makefile                     |    6 +
 drivers/misc/mic/card/Makefile                |   11 +
 drivers/misc/mic/card/mic_debugfs.c           |  130 ++
 drivers/misc/mic/card/mic_device.c            |  306 +++++
 drivers/misc/mic/card/mic_device.h            |  133 ++
 drivers/misc/mic/card/mic_virtio.c            |  631 +++++++++
 drivers/misc/mic/card/mic_virtio.h            |   77 ++
 drivers/misc/mic/card/mic_x100.c              |  256 ++++
 drivers/misc/mic/card/mic_x100.h              |   48 +
 drivers/misc/mic/common/mic_device.h          |   51 +
 drivers/misc/mic/host/Makefile                |   13 +
 drivers/misc/mic/host/mic_boot.c              |  185 +++
 drivers/misc/mic/host/mic_debugfs.c           |  496 +++++++
 drivers/misc/mic/host/mic_device.h            |  300 +++++
 drivers/misc/mic/host/mic_fops.c              |  221 ++++
 drivers/misc/mic/host/mic_fops.h              |   32 +
 drivers/misc/mic/host/mic_main.c              | 1098 ++++++++++++++++
 drivers/misc/mic/host/mic_smpt.c              |  440 +++++++
 drivers/misc/mic/host/mic_smpt.h              |   98 ++
 drivers/misc/mic/host/mic_sysfs.c             |  468 +++++++
 drivers/misc/mic/host/mic_virtio.c            |  704 ++++++++++
 drivers/misc/mic/host/mic_virtio.h            |  138 ++
 drivers/misc/mic/host/mic_x100.c              |  573 +++++++++
 drivers/misc/mic/host/mic_x100.h              |   99 ++
 include/uapi/linux/Kbuild                     |    2 +
 include/uapi/linux/mic_common.h               |  238 ++++
 include/uapi/linux/mic_ioctl.h                |   74 ++
 39 files changed, 9363 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt
 create mode 100644 Documentation/mic/mic_overview.txt
 create mode 100644 Documentation/mic/mpssd/.gitignore
 create mode 100644 Documentation/mic/mpssd/Makefile
 create mode 100755 Documentation/mic/mpssd/micctrl
 create mode 100755 Documentation/mic/mpssd/mpss
 create mode 100644 Documentation/mic/mpssd/mpssd.c
 create mode 100644 Documentation/mic/mpssd/mpssd.h
 create mode 100644 Documentation/mic/mpssd/sysfs.c
 create mode 100644 drivers/misc/mic/Kconfig
 create mode 100644 drivers/misc/mic/Makefile
 create mode 100644 drivers/misc/mic/card/Makefile
 create mode 100644 drivers/misc/mic/card/mic_debugfs.c
 create mode 100644 drivers/misc/mic/card/mic_device.c
 create mode 100644 drivers/misc/mic/card/mic_device.h
 create mode 100644 drivers/misc/mic/card/mic_virtio.c
 create mode 100644 drivers/misc/mic/card/mic_virtio.h
 create mode 100644 drivers/misc/mic/card/mic_x100.c
 create mode 100644 drivers/misc/mic/card/mic_x100.h
 create mode 100644 drivers/misc/mic/common/mic_device.h
 create mode 100644 drivers/misc/mic/host/Makefile
 create mode 100644 drivers/misc/mic/host/mic_boot.c
 create mode 100644 drivers/misc/mic/host/mic_debugfs.c
 create mode 100644 drivers/misc/mic/host/mic_device.h
 create mode 100644 drivers/misc/mic/host/mic_fops.c
 create mode 100644 drivers/misc/mic/host/mic_fops.h
 create mode 100644 drivers/misc/mic/host/mic_main.c
 create mode 100644 drivers/misc/mic/host/mic_smpt.c
 create mode 100644 drivers/misc/mic/host/mic_smpt.h
 create mode 100644 drivers/misc/mic/host/mic_sysfs.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.h
 create mode 100644 drivers/misc/mic/host/mic_x100.c
 create mode 100644 drivers/misc/mic/host/mic_x100.h
 create mode 100644 include/uapi/linux/mic_common.h
 create mode 100644 include/uapi/linux/mic_ioctl.h

Sudeep Dutt

author

2013-09-04 16:19 UTC #35 lore raw

In reply to Sudeep Dutt

Quoted 10 lines · Sudeep Dutt

ChangeLog:
=========
v2 => v3:
a) Patch 1 data structure cleanups, header file include cleanups, IDA interface reuse and switching to device_create_with_groups(..) as per feedback from Greg Kroah-Hartman.
b) Patch 7 signal documentation, sleep workaround removal and sysfs access API cleanups as per feedback from Michael S. Tsirkin.

Quoted 9 lines

v1 => v2: @ http://lwn.net/Articles/563131/
a) License wording cleanup, sysfs ABI documentation, patch 1 refactoring into 3 smaller patches and function renames, as per feedback from Greg Kroah-Hartman.
b) Use VRINGH infrastructure for accessing virtio rings from the host in patch 5, as per feedback from Michael S. Tsirkin.
v1: Initial post @ https://lwn.net/Articles/561314/

Hi Greg,

Thanks in advance,
Sudeep Dutt

Quoted 216 lines

Description:
============
An Intel MIC X100 device is a PCIe form factor add-in coprocessor card based on the Intel Many Integrated Core (MIC) architecture that runs a Linux OS. It is a PCIe endpoint in a platform and therefore implements the three required standard address spaces i.e. configuration, memory and I/O. The host OS loads a device driver as is typical for PCIe devices. The card itself runs a bootstrap after reset that transfers control to the card OS downloaded from the host driver. The card OS as shipped by Intel is a Linux kernel with modifications for the X100 devices.
Since it is a PCIe card, it does not have the ability to host hardware devices for networking, storage and console. We provide these devices on X100 coprocessors thus enabling a self-bootable equivalent environment for applications. A key benefit of our solution is that it leverages the standard virtio framework for network, disk and console devices, though in our case the virtio framework is used across a PCIe bus.
Here is a block diagram of the various components described above. The virtio backends are situated on the host rather than the card given better single threaded performance for the host compared to MIC, the ability of the host to initiate DMA's to/from the card using the MIC DMA engine and the fact that the virtio block storage backend can only be on the host.
       +----------+           |             +----------+
       | Card OS  |           |             | Host OS  |
       +----------+           |             +----------+
                              |
+-------+ +--------+ +------+ | +---------+  +--------+ +--------+
| Virtio| |Virtio  | |Virtio| | |Virtio   |  |Virtio  | |Virtio  |
| Net   | |Console | |Block | | |Net      |  |Console | |Block   |
| Driver| |Driver  | |Driver| | |backend  |  |backend | |backend |
+-------+ +--------+ +------+ | +---------+  +--------+ +--------+
    |         |         |     |      |            |         |
    |         |         |     |User  |            |         |
    |         |         |     |------|------------|---------|-------
    +-------------------+     |Kernel +--------------------------+
              |               |       | Virtio over PCIe IOCTLs  |
              |               |       +--------------------------+
      +--------------+        |                   |
      |Intel MIC     |        |            +---------------+
      |Card Driver   |        |            |Intel MIC      |
      +--------------+        |            |Host Driver    |
              |               |            +---------------+
              |               |                   |
     +-------------------------------------------------------------+
     |                                                             |
     |                    PCIe Bus                                 |
     +-------------------------------------------------------------+
The following series of patches are partitioned as follows:
Patch 1: This patch introduces the "Intel MIC Host Driver" in the block diagram which does the following:
a) Initializes the Intel MIC X100 PCIe devices.
b) Provides sysfs entries for family and stepping information.
Patch 2: This patch enables the following features in the "Intel MIC Host Driver" in the block diagram:
a) MSIx, MSI and legacy interrupt support.
b) System Memory Page Table(SMPT) support. SMPT enables system memory access from the card. On X100 devices the host can program 32 SMPT registers each capable of accessing 16GB of system memory address space from X100 devices. The registers can thereby be used to access a cumulative 512GB of system memory address space from X100 devices at any point in time.
Patch 3: This patch enables the following features in the "Intel MIC Host Driver" in the block diagram:
a) Boots and shuts down the card via sysfs entries.
b) Allocates and maps a device page for communication with the card driver and updates the device page address via scratchpad registers.
c) Provides sysfs entries for shutdown status, kernel command line, firmware, ramdisk, bootmode and log buffer information.
Patch 4: This patch introduces the "Intel MIC Card Driver" in the block diagram which does the following:
a) Initializes the Intel MIC X100 platform device and driver.
b) Sets up support to handle shutdown requests from the host.
c) Maps the device page after obtaining the device page address from the scratchpad registers updated by the host.
d) Informs the host upon a card crash by registering a panic notifier.
e) Informs the host upon a poweroff/halt event.
Patch 5: This patch introduces the host "Virtio over PCIe" interface for Intel MIC. It allows creating user space backends on the host and instantiating virtio devices for them on the Intel MIC card. It uses the existing VRINGH infrastructure in the kernel to access virtio rings from the host. A character device per MIC is exposed with IOCTL, mmap and poll callbacks. This allows the user space backend to:
(a) add/remove a virtio device via a device page.
(b) map (R/O) virtio rings and device page to user space.
(c) poll for availability of data.
(d) copy a descriptor or entire descriptor chain to/from the card.
(e) modify virtio configuration.
(f) handle virtio device reset.
The buffers are copied over using CPU copies for this initial patch and host initiated MIC DMA support is planned for future patches. The avail and desc virtio rings are in host memory and the used ring is in card memory to maximize writes across PCIe for performance.
Patch 6: This patch introduces the card "Virtio over PCIe" interface for Intel MIC. It allows virtio drivers on the card to communicate with their user space backends on the host via a device page. Ring 3 apps on the host can add, remove and configure virtio devices. A thin MIC specific virtio_config_ops is implemented which is borrowed heavily from previous similar implementations in lguest and s390 @ drivers/lguest/lguest_device.c
drivers/s390/kvm/kvm_virtio.c
Patch 7: This patch introduces a sample user space daemon which implements the virtio device backends on the host. The daemon creates/removes/configures virtio device backends by communicating with the Intel MIC Host Driver. The virtio devices currently supported are virtio net, virtio console and virtio block. Virtio net supports TSO/GSO. The daemon also monitors card shutdown status and takes appropriate actions like killing the virtio backends and resetting the card upon card shutdown and crashes.
The patches have been compiled/validated against v3.11-rc5.
Ashutosh Dixit (2):
Intel MIC Host Driver Changes for Virtio Devices. Intel MIC Card Driver Changes for Virtio Devices.
Caz Yokoyama (1):
Sample Implementation of Intel MIC User Space Daemon.
Dasaratharaman Chandramouli (1):
Intel MIC Host Driver Interrupt/SMPT support.
Sudeep Dutt (3):
Intel MIC Host Driver for X100 family. Intel MIC Host Driver, card OS state management. Intel MIC Card Driver for X100 family.
 Documentation/ABI/testing/sysfs-class-mic.txt |  147 +++
 Documentation/mic/mic_overview.txt            |   49 +
 Documentation/mic/mpssd/.gitignore            |    1 +
 Documentation/mic/mpssd/Makefile              |   19 +
 Documentation/mic/mpssd/micctrl               |  173 +++
 Documentation/mic/mpssd/mpss                  |  202 +++
 Documentation/mic/mpssd/mpssd.c               | 1701 +++++++++++++++++++++++++
 Documentation/mic/mpssd/mpssd.h               |  100 ++
 Documentation/mic/mpssd/sysfs.c               |  102 ++
 drivers/misc/Kconfig                          |    1 +
 drivers/misc/Makefile                         |    1 +
 drivers/misc/mic/Kconfig                      |   39 +
 drivers/misc/mic/Makefile                     |    6 +
 drivers/misc/mic/card/Makefile                |   11 +
 drivers/misc/mic/card/mic_debugfs.c           |  130 ++
 drivers/misc/mic/card/mic_device.c            |  306 +++++
 drivers/misc/mic/card/mic_device.h            |  133 ++
 drivers/misc/mic/card/mic_virtio.c            |  631 +++++++++
 drivers/misc/mic/card/mic_virtio.h            |   77 ++
 drivers/misc/mic/card/mic_x100.c              |  256 ++++
 drivers/misc/mic/card/mic_x100.h              |   48 +
 drivers/misc/mic/common/mic_device.h          |   51 +
 drivers/misc/mic/host/Makefile                |   13 +
 drivers/misc/mic/host/mic_boot.c              |  185 +++
 drivers/misc/mic/host/mic_debugfs.c           |  496 +++++++
 drivers/misc/mic/host/mic_device.h            |  300 +++++
 drivers/misc/mic/host/mic_fops.c              |  221 ++++
 drivers/misc/mic/host/mic_fops.h              |   32 +
 drivers/misc/mic/host/mic_main.c              | 1098 ++++++++++++++++
 drivers/misc/mic/host/mic_smpt.c              |  440 +++++++
 drivers/misc/mic/host/mic_smpt.h              |   98 ++
 drivers/misc/mic/host/mic_sysfs.c             |  468 +++++++
 drivers/misc/mic/host/mic_virtio.c            |  704 ++++++++++
 drivers/misc/mic/host/mic_virtio.h            |  138 ++
 drivers/misc/mic/host/mic_x100.c              |  573 +++++++++
 drivers/misc/mic/host/mic_x100.h              |   99 ++
 include/uapi/linux/Kbuild                     |    2 +
 include/uapi/linux/mic_common.h               |  238 ++++
 include/uapi/linux/mic_ioctl.h                |   74 ++
 39 files changed, 9363 insertions(+)
 create mode 100644 Documentation/ABI/testing/sysfs-class-mic.txt
 create mode 100644 Documentation/mic/mic_overview.txt
 create mode 100644 Documentation/mic/mpssd/.gitignore
 create mode 100644 Documentation/mic/mpssd/Makefile
 create mode 100755 Documentation/mic/mpssd/micctrl
 create mode 100755 Documentation/mic/mpssd/mpss
 create mode 100644 Documentation/mic/mpssd/mpssd.c
 create mode 100644 Documentation/mic/mpssd/mpssd.h
 create mode 100644 Documentation/mic/mpssd/sysfs.c
 create mode 100644 drivers/misc/mic/Kconfig
 create mode 100644 drivers/misc/mic/Makefile
 create mode 100644 drivers/misc/mic/card/Makefile
 create mode 100644 drivers/misc/mic/card/mic_debugfs.c
 create mode 100644 drivers/misc/mic/card/mic_device.c
 create mode 100644 drivers/misc/mic/card/mic_device.h
 create mode 100644 drivers/misc/mic/card/mic_virtio.c
 create mode 100644 drivers/misc/mic/card/mic_virtio.h
 create mode 100644 drivers/misc/mic/card/mic_x100.c
 create mode 100644 drivers/misc/mic/card/mic_x100.h
 create mode 100644 drivers/misc/mic/common/mic_device.h
 create mode 100644 drivers/misc/mic/host/Makefile
 create mode 100644 drivers/misc/mic/host/mic_boot.c
 create mode 100644 drivers/misc/mic/host/mic_debugfs.c
 create mode 100644 drivers/misc/mic/host/mic_device.h
 create mode 100644 drivers/misc/mic/host/mic_fops.c
 create mode 100644 drivers/misc/mic/host/mic_fops.h
 create mode 100644 drivers/misc/mic/host/mic_main.c
 create mode 100644 drivers/misc/mic/host/mic_smpt.c
 create mode 100644 drivers/misc/mic/host/mic_smpt.h
 create mode 100644 drivers/misc/mic/host/mic_sysfs.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.c
 create mode 100644 drivers/misc/mic/host/mic_virtio.h
 create mode 100644 drivers/misc/mic/host/mic_x100.c
 create mode 100644 drivers/misc/mic/host/mic_x100.h
 create mode 100644 include/uapi/linux/mic_common.h
 create mode 100644 include/uapi/linux/mic_ioctl.h

Greg Kroah-Hartman

maintainer

2013-09-04 16:32 UTC #36 lore raw

Quoted 21 lines · Sudeep Dutt

Quoted 10 lines · Sudeep Dutt
ChangeLog:
=========
v2 => v3:
a) Patch 1 data structure cleanups, header file include cleanups, IDA interface reuse and switching to device_create_with_groups(..) as per feedback from Greg Kroah-Hartman.
b) Patch 7 signal documentation, sleep workaround removal and sysfs access API cleanups as per feedback from Michael S. Tsirkin.
Quoted 9 lines
v1 => v2: @ http://lwn.net/Articles/563131/
a) License wording cleanup, sysfs ABI documentation, patch 1 refactoring into 3 smaller patches and function renames, as per feedback from Greg Kroah-Hartman.
b) Use VRINGH infrastructure for accessing virtio rings from the host in patch 5, as per feedback from Michael S. Tsirkin.
v1: Initial post @ https://lwn.net/Articles/561314/

Quoted 6 lines

Hi Greg,
It has been a couple of weeks since we posted v3 @
https://lwn.net/Articles/564795 which addresses all feedback received so far. Have you had a chance to review this version? Please let us know your thoughts.

I don't have this around anymore, I got 7 copies and thought your patch-send script went crazy so I deleted them :(

Care to send them again? But note, this is the merge window, so I'll not be able to do anythign for at least 2 weeks with them.

thanks,

greg k-h