Skip to contentSkip to conversation

Enable DSA 2.0 Event Log and completion record faulting features

PATCH Superseded Fenghua Yu 4 participants 25 messages Mar 6, 2023 – Mar 31, 2023

Subsystems (3)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Fenghua Yu

author
#0 lore raw

v2 cover Enable DSA 2.0 Event Log and completion record faulting features

Applications can send 64B descriptors to the DSA device via CPU instructions MOVDIR64B or ENQCMD. The application can choose to have the device write back a completion record (CR) in system memory to indicate the status of the descriptor submitted on completion.

With the DSA hardware, the device is able to do on demand paging through the hardware by faulting in the user pages that do not have physical memory page backing with assistance from IOMMU. In the spec this was designated as the block on fault feature. While this hardware feature made operation simpler, it also stalls the device engines while the memory pages are being faulted in through Page Request Service (PRS). For applications sharing the same workqueue (wq) or wqs in the same group, operations are stalled if there are no free engines. To avoid slowing the performance of all other running applications sharing the same device engine(s), PRS can to be disabled and software can deal with partial completion.

The block on fault feature on DSA 1.0 can be disabled for the wq. However, PRS is not completely disabled for the whole path. It is not disabled for CRs or batch list for a batch operation.

The other issue is the DSA 1.0 error reporting mechanism, SWERROR register. The SWERROR register can only report a single error at a time until the driver reads and acknowledges the error. The follow on errors cannot be reported until the current error is "cleared" by the software by writing a bit to the SWERR register. If a large number of faults arrive and the software cannot clear them fast enough, overflowed errors will be dropped by the device.

A CR is the optional 32 bytes (DSA) or 64 bytes (IAA) status that is written back for a submitted descriptor. If the address for the CR faults, the error is reported to the SWERROR register instead.

With DSA 2.0 hardware [1], the event log feature is added. All errors are reported as an entry in a circular buffer reside in the system memory. The system admin is responsible to configure the size of the circular buffer large enough per device to handle the potential errors that may be reported. If the buffer is full and another error needs to be reported, the device engine will block until there's a free slot in the buffer. An event log entry for a faulted CR will contain the error information, the CR address that faulted, and the expected CR content the device had originally intended to write.

DSA 2.0 also introduces per wq PRS disable knob. This will disable all PRS operations for the specific wq. The device will still have Address Translation Service (ATS) on. When ATS fails on a memory address for a CR, an eventlog entry will be written by the hardware into the event log ring buffer. The driver software is expected to parse the event log entry, fault in the address of the CR, and the write the content of the CR to the memory address.

This patch series will implement the DSA 2 event log support. The support for the handling of the faulted user CR is added. The driver is also adding the same support for batch operation descriptors. With a batch operation the handling of the event log entry is a bit more complex. The faulting CR could be for the batch descriptor or any of the operation descriptors within the batch. The hardware generates a batch identifier that is used by the driver software to correlate the event log entries for the relevant descriptors of that batch.

The faulting of source and destination addresses for the operation is not handled by the driver. That is left to be handled by the user application by faulting in the memory and re-submit the remaining operation.

This series consists of three parts:
1. Patch 1: Make misc interrupt one shot. Event Log interrupt depends on this patch. This patch was released before but is not in upstream yet:
https://lore.kernel.org/dmaengine/165125374675.311834.10460196228320964350.stgit@djiang5-desk3.ch.intel.com/
2. Patches 2-15: Enable Event Log and Completion Record faulting.
3. Patch 16: Configure PRS disable per WQ.

This series is applied cleanly on top of "Expose IAA 2.0 device capabilities" series:
https://lore.kernel.org/lkml/[email protected]/

Change log:
v2:
- Define and export iommu_access_remote_vm() for IDXD driver to write completion record to user address space. This change removes patch 8 and 9 in v1 (Alistair Popple)

Dave Jiang (15):
dmaengine: idxd: make misc interrupt one shot
dmaengine: idxd: add event log size sysfs attribute
dmaengine: idxd: setup event log configuration
dmaengine: idxd: add interrupt handling for event log
dmanegine: idxd: add debugfs for event log dump
dmaengine: idxd: add per DSA wq workqueue for processing cr faults
dmaengine: idxd: create kmem cache for event log fault items
dmaengine: idxd: process user page faults for completion record
dmaengine: idxd: add descs_completed field for completion record
dmaengine: idxd: process batch descriptor completion record faults
dmaengine: idxd: add per file user counters for completion record
faults
dmaengine: idxd: add a device to represent the file opened
dmaengine: idxd: expose fault counters to sysfs
dmaengine: idxd: add pid to exported sysfs attribute for opened file
dmaengine: idxd: add per wq PRS disable

Fenghua Yu (1):
iommu: define and export iommu_access_remote_vm()

 .../ABI/stable/sysfs-driver-dma-idxd          |  43 +++
 drivers/dma/Kconfig                           |   1 +
 drivers/dma/idxd/Makefile                     |   2 +-
 drivers/dma/idxd/cdev.c                       | 264 ++++++++++++++++--
 drivers/dma/idxd/debugfs.c                    | 138 +++++++++
 drivers/dma/idxd/device.c                     | 113 +++++++-
 drivers/dma/idxd/idxd.h                       |  63 +++++
 drivers/dma/idxd/init.c                       |  53 ++++
 drivers/dma/idxd/irq.c                        | 203 ++++++++++++--
 drivers/dma/idxd/registers.h                  | 105 ++++++-
 drivers/dma/idxd/sysfs.c                      | 112 +++++++-
 drivers/iommu/iommu-sva.c                     |  35 +++
 include/linux/iommu.h                         |   9 +
 include/uapi/linux/idxd.h                     |  15 +-
 14 files changed, 1091 insertions(+), 65 deletions(-)
 create mode 100644 drivers/dma/idxd/debugfs.c
Signature · 1 line
2.37.1
author
#1 lore raw

PATCH v2 1/16 dmaengine: idxd: make misc interrupt one shot

From: Dave Jiang <[email protected]>

Current code continuously processes the interrupt as long as the hardware is setting the status bit. There's no reason to do that since the threaded handler will get called again if another interrupt is asserted.

Also through testing, it has shown that if a misprogrammed (or malicious) agent can continuously submit descriptors with bad completion record and causes errors to be reported via the misc interrupt. Continuous processing by the thread can cause software hang watchdog to kick off since the thread isn't giving up the CPU.

---
 drivers/dma/idxd/irq.c | 38 ++++++++++++--------------------------
 1 file changed, 12 insertions(+), 26 deletions(-)
drivers/dma/idxd/irq.c +12 -26
index aa314ebec587..0d639303b515 100644
@@ -217,13 +217,22 @@ static void idxd_int_handle_revoke(struct work_struct *work)
 	kfree(revoke);
 }
 
-static int process_misc_interrupts(struct idxd_device *idxd, u32 cause)
+irqreturn_t idxd_misc_thread(int vec, void *data)
 {
+	struct idxd_irq_entry *irq_entry = data;
+	struct idxd_device *idxd = ie_to_idxd(irq_entry);
 	struct device *dev = &idxd->pdev->dev;
 	union gensts_reg gensts;
 	u32 val = 0;
 	int i;
 	bool err = false;
+	u32 cause;
+
+	cause = ioread32(idxd->reg_base + IDXD_INTCAUSE_OFFSET);
+	if (!cause)
+		return IRQ_NONE;
+
+	iowrite32(cause, idxd->reg_base + IDXD_INTCAUSE_OFFSET);
 
 	if (cause & IDXD_INTC_HALT_STATE)
 		goto halt;
@@ -301,7 +310,7 @@ static int process_misc_interrupts(struct idxd_device *idxd, u32 cause)
 			      val);
 
 	if (!err)
-		return 0;
+		goto out;
 
 halt:
 	gensts.bits = ioread32(idxd->reg_base + IDXD_GENSTATS_OFFSET);
@@ -324,33 +333,10 @@ static int process_misc_interrupts(struct idxd_device *idxd, u32 cause)
 				"idxd halted, need %s.\n",
 				gensts.reset_type == IDXD_DEVICE_RESET_FLR ?
 				"FLR" : "system reset");
-			return -ENXIO;
 		}
 	}
 
-	return 0;
-}
-
-irqreturn_t idxd_misc_thread(int vec, void *data)
-{
-	struct idxd_irq_entry *irq_entry = data;
-	struct idxd_device *idxd = ie_to_idxd(irq_entry);
-	int rc;
-	u32 cause;
-
-	cause = ioread32(idxd->reg_base + IDXD_INTCAUSE_OFFSET);
-	if (cause)
-		iowrite32(cause, idxd->reg_base + IDXD_INTCAUSE_OFFSET);
-
-	while (cause) {
-		rc = process_misc_interrupts(idxd, cause);
-		if (rc < 0)
-			break;
-		cause = ioread32(idxd->reg_base + IDXD_INTCAUSE_OFFSET);
-		if (cause)
-			iowrite32(cause, idxd->reg_base + IDXD_INTCAUSE_OFFSET);
-	}
-
+out:
 	return IRQ_HANDLED;
 }
 
Signature · 1 line
2.37.1
author
#2 lore raw

PATCH v2 2/16 dmaengine: idxd: add event log size sysfs attribute

In reply to Fenghua Yu

From: Dave Jiang <[email protected]>

Add support for changing of the event log size. Event log is a feature added to DSA 2.0 hardware to improve error reporting. It supersedes the SWERROR register on DSA 1.0 hardware and hope to prevent loss of reported errors.

The error log size determines how many error entries supported for the device. It can be configured by the user via sysfs attribute.

---
 .../ABI/stable/sysfs-driver-dma-idxd          |  8 +++
 drivers/dma/idxd/idxd.h                       |  5 ++
 drivers/dma/idxd/init.c                       | 23 ++++++++
 drivers/dma/idxd/registers.h                  |  7 ++-
 drivers/dma/idxd/sysfs.c                      | 52 +++++++++++++++++++
 5 files changed, 94 insertions(+), 1 deletion(-)
Documentation/ABI/stable/sysfs-driver-dma-idxd +8 -0
index d5e3dd3d8434..e01916611452 100644
@@ -144,6 +144,14 @@ Description:	IAA (IAX) capability mask. Exported to user space for application
 		consumption. This attribute should only be visible on IAA devices
 		that are version 2 or later.
 
+What:		/sys/bus/dsa/devices/dsa<m>/event_log_size
+Date:		Sept 14, 2022
+KernelVersion: 6.4.0
+Contact:	[email protected]
+Description:	The event log size to be configured. Default is 64 entries and
+		occupies 4k size if the evl entry is 64 bytes. It's visible
+		only on platforms that support the capability.
+
 What:		/sys/bus/dsa/devices/wq<m>.<n>/block_on_fault
 Date:		Oct 27, 2020
 KernelVersion:	5.11.0
drivers/dma/idxd/idxd.h +5 -0
index eca2c9d76db6..2a71273f1822 100644
@@ -261,6 +261,10 @@ struct idxd_driver_data {
 	int align;
 };
 
+struct idxd_evl {
+	u16 size;
+};
+
 struct idxd_device {
 	struct idxd_dev idxd_dev;
 	struct idxd_driver_data *data;
@@ -317,6 +321,7 @@ struct idxd_device {
 	struct idxd_pmu *idxd_pmu;
 
 	unsigned long *opcap_bmap;
+	struct idxd_evl *evl;
 };
 
 /* IDXD software descriptor */
drivers/dma/idxd/init.c +23 -0
index a408fc91144d..d1fb01c115d8 100644
@@ -332,6 +332,23 @@ static void idxd_cleanup_internals(struct idxd_device *idxd)
 	destroy_workqueue(idxd->wq);
 }
 
+static int idxd_init_evl(struct idxd_device *idxd)
+{
+	struct device *dev = &idxd->pdev->dev;
+	struct idxd_evl *evl;
+
+	if (idxd->hw.gen_cap.evl_support == 0)
+		return 0;
+
+	evl = kzalloc_node(sizeof(*evl), GFP_KERNEL, dev_to_node(dev));
+	if (!evl)
+		return -ENOMEM;
+
+	evl->size = IDXD_EVL_SIZE_MIN;
+	idxd->evl = evl;
+	return 0;
+}
+
 static int idxd_setup_internals(struct idxd_device *idxd)
 {
 	struct device *dev = &idxd->pdev->dev;
@@ -357,8 +374,14 @@ static int idxd_setup_internals(struct idxd_device *idxd)
 		goto err_wkq_create;
 	}
 
+	rc = idxd_init_evl(idxd);
+	if (rc < 0)
+		goto err_evl;
+
 	return 0;
 
+ err_evl:
+	destroy_workqueue(idxd->wq);
  err_wkq_create:
 	for (i = 0; i < idxd->max_groups; i++)
 		put_device(group_confdev(idxd->groups[i]));
drivers/dma/idxd/registers.h +6 -1
index 338289a66f00..ea3a499a3c3c 100644
@@ -31,7 +31,9 @@ union gen_cap_reg {
 		u64 rsvd:3;
 		u64 dest_readback:1;
 		u64 drain_readback:1;
-		u64 rsvd2:6;
+		u64 rsvd2:3;
+		u64 evl_support:2;
+		u64 rsvd4:1;
 		u64 max_xfer_shift:5;
 		u64 max_batch_shift:4;
 		u64 max_ims_mult:6;
@@ -297,6 +299,9 @@ union iaa_cap_reg {
 
 #define IDXD_IAACAP_OFFSET	0x180
 
+#define IDXD_EVL_SIZE_MIN	0x0040
+#define IDXD_EVL_SIZE_MAX	0xffff
+
 union msix_perm {
 	struct {
 		u32 rsvd:2;
drivers/dma/idxd/sysfs.c +52 -0

82 lines not on this page · raw message

Signature · 1 line
2.37.1
author
#3 lore raw

PATCH v2 3/16 dmaengine: idxd: setup event log configuration

In reply to Fenghua Yu

From: Dave Jiang <[email protected]>

Add setup of event log feature for supported device. Event log addresses error reporting that was lacking in gen 1 DSA devices where a second error event does not get reported when a first event is pending software handling. The event log allows a circular buffer that the device can push error events to. It is up to the user to create a large enough event log ring in order to capture the expected events. The evl size can be set in the device sysfs attribute. By default 64 entries are supported as minimal when event log is enabled.

---
 drivers/dma/idxd/device.c    | 89 +++++++++++++++++++++++++++++++++++-
 drivers/dma/idxd/idxd.h      | 19 ++++++++
 drivers/dma/idxd/init.c      |  1 +
 drivers/dma/idxd/registers.h | 72 ++++++++++++++++++++++++++++-
 drivers/dma/idxd/sysfs.c     |  3 +-
 include/uapi/linux/idxd.h    |  1 +
 6 files changed, 181 insertions(+), 4 deletions(-)
drivers/dma/idxd/device.c +88 -1
index 5f321f3b4242..230fe9bb56ae 100644
@@ -752,6 +752,83 @@ void idxd_device_clear_state(struct idxd_device *idxd)
 	spin_unlock(&idxd->dev_lock);
 }
 
+static int idxd_device_evl_setup(struct idxd_device *idxd)
+{
+	union gencfg_reg gencfg;
+	union evlcfg_reg evlcfg;
+	union genctrl_reg genctrl;
+	struct device *dev = &idxd->pdev->dev;
+	void *addr;
+	dma_addr_t dma_addr;
+	int size;
+	struct idxd_evl *evl = idxd->evl;
+
+	if (!evl)
+		return 0;
+
+	size = evl_size(idxd);
+	/*
+	 * Address needs to be page aligned. However, dma_alloc_coherent() provides
+	 * at minimal page size aligned address. No manual alignment required.
+	 */
+	addr = dma_alloc_coherent(dev, size, &dma_addr, GFP_KERNEL);
+	if (!addr)
+		return -ENOMEM;
+
+	memset(addr, 0, size);
+
+	spin_lock(&evl->lock);
+	evl->log = addr;
+	evl->dma = dma_addr;
+	evl->log_size = size;
+
+	memset(&evlcfg, 0, sizeof(evlcfg));
+	evlcfg.bits[0] = dma_addr & GENMASK(63, 12);
+	evlcfg.size = evl->size;
+
+	iowrite64(evlcfg.bits[0], idxd->reg_base + IDXD_EVLCFG_OFFSET);
+	iowrite64(evlcfg.bits[1], idxd->reg_base + IDXD_EVLCFG_OFFSET + 8);
+
+	genctrl.bits = ioread32(idxd->reg_base + IDXD_GENCTRL_OFFSET);
+	genctrl.evl_int_en = 1;
+	iowrite32(genctrl.bits, idxd->reg_base + IDXD_GENCTRL_OFFSET);
+
+	gencfg.bits = ioread32(idxd->reg_base + IDXD_GENCFG_OFFSET);
+	gencfg.evl_en = 1;
+	iowrite32(gencfg.bits, idxd->reg_base + IDXD_GENCFG_OFFSET);
+
+	spin_unlock(&evl->lock);
+	return 0;
+}
+
+static void idxd_device_evl_free(struct idxd_device *idxd)
+{
+	union gencfg_reg gencfg;
+	union genctrl_reg genctrl;
+	struct device *dev = &idxd->pdev->dev;
+	struct idxd_evl *evl = idxd->evl;
+
+	gencfg.bits = ioread32(idxd->reg_base + IDXD_GENCFG_OFFSET);
+	if (!gencfg.evl_en)
+		return;
+
+	spin_lock(&evl->lock);
+	gencfg.evl_en = 0;
+	iowrite32(gencfg.bits, idxd->reg_base + IDXD_GENCFG_OFFSET);
+
+	genctrl.bits = ioread32(idxd->reg_base + IDXD_GENCTRL_OFFSET);
+	genctrl.evl_int_en = 0;
+	iowrite32(genctrl.bits, idxd->reg_base + IDXD_GENCTRL_OFFSET);
+
+	iowrite64(0, idxd->reg_base + IDXD_EVLCFG_OFFSET);
+	iowrite64(0, idxd->reg_base + IDXD_EVLCFG_OFFSET + 8);
+
+	dma_free_coherent(dev, evl->log_size, evl->log, evl->dma);
+	evl->log = NULL;
+	evl->size = IDXD_EVL_SIZE_MIN;
+	spin_unlock(&evl->lock);
+}
+
 static void idxd_group_config_write(struct idxd_group *group)
 {
 	struct idxd_device *idxd = group->idxd;
@@ -1451,15 +1528,24 @@ int idxd_device_drv_probe(struct idxd_dev *idxd_dev)
 	if (rc < 0)
 		return -ENXIO;
 
+	rc = idxd_device_evl_setup(idxd);
+	if (rc < 0) {
+		idxd->cmd_status = IDXD_SCMD_DEV_EVL_ERR;
+		return rc;
+	}
+
 	/* Start device */
 	rc = idxd_device_enable(idxd);
-	if (rc < 0)
+	if (rc < 0) {
+		idxd_device_evl_free(idxd);
 		return rc;
+	}
 
 	/* Setup DMA device without channels */
 	rc = idxd_register_dma_device(idxd);
 	if (rc < 0) {
 		idxd_device_disable(idxd);
+		idxd_device_evl_free(idxd);
 		idxd->cmd_status = IDXD_SCMD_DEV_DMA_ERR;
 		return rc;
 	}
@@ -1488,6 +1574,7 @@ void idxd_device_drv_remove(struct idxd_dev *idxd_dev)
 	idxd_device_disable(idxd);
 	if (test_bit(IDXD_FLAG_CONFIGURABLE, &idxd->flags))
 		idxd_device_reset(idxd);
+	idxd_device_evl_free(idxd);
 }
 
 static enum idxd_dev_type dev_types[] = {
drivers/dma/idxd/idxd.h +19 -0
index 2a71273f1822..c74681f02b18 100644
@@ -262,7 +262,15 @@ struct idxd_driver_data {
 };
 
 struct idxd_evl {
+	/* Lock to protect event log access. */
+	spinlock_t lock;
+	void *log;
+	dma_addr_t dma;
+	/* Total size of event log = number of entries * entry size. */
+	unsigned int log_size;
+	/* The number of entries in the event log. */
 	u16 size;
+	u16 head;
 };
 
 struct idxd_device {
@@ -324,6 +332,17 @@ struct idxd_device {
 	struct idxd_evl *evl;
 };
 
+static inline unsigned int evl_ent_size(struct idxd_device *idxd)
+{
+	return idxd->hw.gen_cap.evl_support ?
+	       (32 * (1 << idxd->hw.gen_cap.evl_support)) : 0;
+}
+
+static inline unsigned int evl_size(struct idxd_device *idxd)
+{
+	return idxd->evl->size * evl_ent_size(idxd);
+}
+
 /* IDXD software descriptor */
 struct idxd_desc {
 	union {
drivers/dma/idxd/init.c +1 -0
index d1fb01c115d8..2ffeb2f3a2c8 100644
@@ -344,6 +344,7 @@ static int idxd_init_evl(struct idxd_device *idxd)
 	if (!evl)
 		return -ENOMEM;
 
+	spin_lock_init(&evl->lock);
 	evl->size = IDXD_EVL_SIZE_MIN;
 	idxd->evl = evl;
 	return 0;
drivers/dma/idxd/registers.h +70 -2

100 lines not on this page · raw message

drivers/dma/idxd/sysfs.c +2 -1
index 85644e5bde83..163fdfaa5022 100644
@@ -1605,7 +1605,8 @@ static ssize_t event_log_size_store(struct device *dev,
 	if (!test_bit(IDXD_FLAG_CONFIGURABLE, &idxd->flags))
 		return -EPERM;
 
-	if (val < IDXD_EVL_SIZE_MIN || val > IDXD_EVL_SIZE_MAX)
+	if (val < IDXD_EVL_SIZE_MIN || val > IDXD_EVL_SIZE_MAX ||
+	    (val * evl_ent_size(idxd) > ULONG_MAX - idxd->evl->dma))
 		return -EINVAL;
 
 	idxd->evl->size = val;
include/uapi/linux/idxd.h +1 -0
index fc47635b57dc..5d05bf12f2bd 100644
@@ -30,6 +30,7 @@ enum idxd_scmd_stat {
 	IDXD_SCMD_WQ_NO_PRIV = 0x800f0000,
 	IDXD_SCMD_WQ_IRQ_ERR = 0x80100000,
 	IDXD_SCMD_WQ_USER_NO_IOMMU = 0x80110000,
+	IDXD_SCMD_DEV_EVL_ERR = 0x80120000,
 };
 
 #define IDXD_SCMD_SOFTERR_MASK	0x80000000
Signature · 1 line
2.37.1
author
#4 lore raw

PATCH v2 4/16 dmaengine: idxd: add interrupt handling for event log

In reply to Fenghua Yu

From: Dave Jiang <[email protected]>

An event log interrupt is raised in the misc interrupt INTCAUSE register when an event is written by the hardware. Add basic event log processing support to the interrupt handler. The event log is a ring where the hardware owns the tail and the software owns the head. The hardware will advance the tail index when an additional event has been pushed to memory. The software will process the log entry and then advances the head. The log is full when (tail + 1) % log_size = head. The hardware will stop writing when the log is full. The user is expected to create a log size large enough to handle all the expected events.

---
 drivers/dma/idxd/irq.c       | 48 ++++++++++++++++++++++++++++++++++++
 drivers/dma/idxd/registers.h | 19 ++++++++++++++
 include/uapi/linux/idxd.h    |  1 +
 3 files changed, 68 insertions(+)
drivers/dma/idxd/irq.c +48 -0
index 0d639303b515..52b8b7d9db22 100644
@@ -217,6 +217,49 @@ static void idxd_int_handle_revoke(struct work_struct *work)
 	kfree(revoke);
 }
 
+static void process_evl_entry(struct idxd_device *idxd, struct __evl_entry *entry_head)
+{
+	struct device *dev = &idxd->pdev->dev;
+	u8 status;
+
+	status = DSA_COMP_STATUS(entry_head->error);
+	dev_warn_ratelimited(dev, "Device error %#x operation: %#x fault addr: %#llx\n",
+			     status, entry_head->operation, entry_head->fault_addr);
+}
+
+static void process_evl_entries(struct idxd_device *idxd)
+{
+	union evl_status_reg evl_status;
+	unsigned int h, t;
+	struct idxd_evl *evl = idxd->evl;
+	struct __evl_entry *entry_head;
+	unsigned int ent_size = evl_ent_size(idxd);
+	u32 size;
+
+	evl_status.bits = 0;
+	evl_status.int_pending = 1;
+
+	spin_lock(&evl->lock);
+	/* Clear interrupt pending bit */
+	iowrite32(evl_status.bits_upper32,
+		  idxd->reg_base + IDXD_EVLSTATUS_OFFSET + sizeof(u32));
+	h = evl->head;
+	evl_status.bits = ioread64(idxd->reg_base + IDXD_EVLSTATUS_OFFSET);
+	t = evl_status.tail;
+	size = idxd->evl->size;
+
+	while (h != t) {
+		entry_head = (struct __evl_entry *)(evl->log + (h * ent_size));
+		process_evl_entry(idxd, entry_head);
+		h = (h + 1) % size;
+	}
+
+	evl->head = h;
+	evl_status.head = h;
+	iowrite32(evl_status.bits_lower32, idxd->reg_base + IDXD_EVLSTATUS_OFFSET);
+	spin_unlock(&evl->lock);
+}
+
 irqreturn_t idxd_misc_thread(int vec, void *data)
 {
 	struct idxd_irq_entry *irq_entry = data;
@@ -304,6 +347,11 @@ irqreturn_t idxd_misc_thread(int vec, void *data)
 		perfmon_counter_overflow(idxd);
 	}
 
+	if (cause & IDXD_INTC_EVL) {
+		val |= IDXD_INTC_EVL;
+		process_evl_entries(idxd);
+	}
+
 	val ^= cause;
 	if (val)
 		dev_warn_once(dev, "Unexpected interrupt cause bits set: %#x\n",
drivers/dma/idxd/registers.h +19 -0
index 11bb97cf7481..148db94f9373 100644
@@ -168,6 +168,7 @@ enum idxd_device_reset_type {
 #define IDXD_INTC_OCCUPY			0x04
 #define IDXD_INTC_PERFMON_OVFL		0x08
 #define IDXD_INTC_HALT_STATE		0x10
+#define IDXD_INTC_EVL			0x20
 #define IDXD_INTC_INT_HANDLE_REVOKED	0x80000000
 
 #define IDXD_CMD_OFFSET			0xa0
@@ -558,6 +559,24 @@ union filter_cfg {
 	u64 val;
 } __packed;
 
+#define IDXD_EVLSTATUS_OFFSET		0xf0
+
+union evl_status_reg {
+	struct {
+		u32 head:16;
+		u32 rsvd:16;
+		u32 tail:16;
+		u32 rsvd2:14;
+		u32 int_pending:1;
+		u32 rsvd3:1;
+	};
+	struct {
+		u32 bits_lower32;
+		u32 bits_upper32;
+	};
+	u64 bits;
+} __packed;
+
 struct __evl_entry {
 	u64 rsvd:2;
 	u64 desc_valid:1;
include/uapi/linux/idxd.h +1 -0
index 5d05bf12f2bd..0bc8eea18586 100644
@@ -170,6 +170,7 @@ enum iax_completion_status {
 
 #define DSA_COMP_STATUS_MASK		0x7f
 #define DSA_COMP_STATUS_WRITE		0x80
+#define DSA_COMP_STATUS(status)		((status) & DSA_COMP_STATUS_MASK)
 
 struct dsa_hw_desc {
 	uint32_t	pasid:20;
Signature · 1 line
2.37.1
author
#5 lore raw

PATCH v2 5/16 dmanegine: idxd: add debugfs for event log dump

In reply to Fenghua Yu

From: Dave Jiang <[email protected]>

Add debugfs entry to dump the content of the event log for debugging. The function will dump all non-zero entries in the event log. It will note which entries are processed and which entries are still pending processing at the time of the dump. The entries may not always be in chronological order due to the log is a circular buffer.

---
 drivers/dma/idxd/Makefile  |   2 +-
 drivers/dma/idxd/debugfs.c | 138 +++++++++++++++++++++++++++++++++++++
 drivers/dma/idxd/idxd.h    |   9 +++
 drivers/dma/idxd/init.c    |  12 ++++
 include/uapi/linux/idxd.h  |   6 +-
 5 files changed, 164 insertions(+), 3 deletions(-)
 create mode 100644 drivers/dma/idxd/debugfs.c
drivers/dma/idxd/Makefile +1 -1
index a1e9f2b3a37c..dc096839ac63 100644
@@ -1,7 +1,7 @@
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=IDXD
 
 obj-$(CONFIG_INTEL_IDXD) += idxd.o
-idxd-y := init.o irq.o device.o sysfs.o submit.o dma.o cdev.o
+idxd-y := init.o irq.o device.o sysfs.o submit.o dma.o cdev.o debugfs.o
 
 idxd-$(CONFIG_INTEL_IDXD_PERFMON) += perfmon.o
 
drivers/dma/idxd/debugfs.c new file +138 -0

138 lines not on this page · raw message

drivers/dma/idxd/idxd.h +9 -0
index c74681f02b18..b923b90b7299 100644
@@ -330,6 +330,9 @@ struct idxd_device {
 
 	unsigned long *opcap_bmap;
 	struct idxd_evl *evl;
+
+	struct dentry *dbgfs_dir;
+	struct dentry *dbgfs_evl_file;
 };
 
 static inline unsigned int evl_ent_size(struct idxd_device *idxd)
@@ -704,4 +707,10 @@ static inline void perfmon_init(void) {}
 static inline void perfmon_exit(void) {}
 #endif
 
+/* debugfs */
+int idxd_device_init_debugfs(struct idxd_device *idxd);
+void idxd_device_remove_debugfs(struct idxd_device *idxd);
+int idxd_init_debugfs(void);
+void idxd_remove_debugfs(void);
+
 #endif
drivers/dma/idxd/init.c +12 -0
index 2ffeb2f3a2c8..d19bc6389221 100644
@@ -670,6 +670,10 @@ static int idxd_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		goto err_dev_register;
 	}
 
+	rc = idxd_device_init_debugfs(idxd);
+	if (rc)
+		dev_warn(dev, "IDXD debugfs failed to setup\n");
+
 	dev_info(&pdev->dev, "Intel(R) Accelerator Device (v%x)\n",
 		 idxd->hw.version);
 
@@ -732,6 +736,7 @@ static void idxd_remove(struct pci_dev *pdev)
 	idxd_shutdown(pdev);
 	if (device_pasid_enabled(idxd))
 		idxd_disable_system_pasid(idxd);
+	idxd_device_remove_debugfs(idxd);
 
 	irq_entry = idxd_get_ie(idxd, 0);
 	free_irq(irq_entry->vector, irq_entry);
@@ -789,6 +794,10 @@ static int __init idxd_init_module(void)
 	if (err)
 		goto err_cdev_register;
 
+	err = idxd_init_debugfs();
+	if (err)
+		goto err_debugfs;
+
 	err = pci_register_driver(&idxd_pci_driver);
 	if (err)
 		goto err_pci_register;
@@ -796,6 +805,8 @@ static int __init idxd_init_module(void)
 	return 0;
 
 err_pci_register:
+	idxd_remove_debugfs();
+err_debugfs:
 	idxd_cdev_remove();
 err_cdev_register:
 	idxd_driver_unregister(&idxd_user_drv);
@@ -816,5 +827,6 @@ static void __exit idxd_exit_module(void)
 	pci_unregister_driver(&idxd_pci_driver);
 	idxd_cdev_remove();
 	perfmon_exit();
+	idxd_remove_debugfs();
 }
 module_exit(idxd_exit_module);
include/uapi/linux/idxd.h +4 -2
index 0bc8eea18586..e86199d09a91 100644
@@ -311,7 +311,8 @@ struct dsa_completion_record {
 		uint8_t		result;
 		uint8_t		dif_status;
 	};
-	uint16_t		rsvd;
+	uint8_t			fault_info;
+	uint8_t			rsvd;
 	uint32_t		bytes_completed;
 	uint64_t		fault_addr;
 	union {
@@ -368,7 +369,8 @@ struct dsa_raw_completion_record {
 struct iax_completion_record {
 	volatile uint8_t        status;
 	uint8_t                 error_code;
-	uint16_t                rsvd;
+	uint8_t			fault_info;
+	uint8_t			rsvd;
 	uint32_t                bytes_completed;
 	uint64_t                fault_addr;
 	uint32_t                invalid_flags;
Signature · 1 line
2.37.1
author
#6 lore raw

PATCH v2 6/16 dmaengine: idxd: add per DSA wq workqueue for processing cr faults

In reply to Fenghua Yu

From: Dave Jiang <[email protected]>

Add a workqueue for user submitted completion record fault processing. The workqueue creation and destruction lifetime will be tied to the user sub-driver since it will only be used when the wq is a user type.

---
 drivers/dma/idxd/cdev.c | 11 +++++++++++
 drivers/dma/idxd/idxd.h |  1 +
 2 files changed, 12 insertions(+)
drivers/dma/idxd/cdev.c +11 -0
index 674bfefca088..cbe29e1a6a44 100644
@@ -330,6 +330,13 @@ static int idxd_user_drv_probe(struct idxd_dev *idxd_dev)
 	}
 
 	mutex_lock(&wq->wq_lock);
+
+	wq->wq = create_workqueue(dev_name(wq_confdev(wq)));
+	if (!wq->wq) {
+		rc = -ENOMEM;
+		goto wq_err;
+	}
+
 	wq->type = IDXD_WQT_USER;
 	rc = drv_enable_wq(wq);
 	if (rc < 0)
@@ -348,7 +355,9 @@ static int idxd_user_drv_probe(struct idxd_dev *idxd_dev)
 err_cdev:
 	drv_disable_wq(wq);
 err:
+	destroy_workqueue(wq->wq);
 	wq->type = IDXD_WQT_NONE;
+wq_err:
 	mutex_unlock(&wq->wq_lock);
 	return rc;
 }
@@ -361,6 +370,8 @@ static void idxd_user_drv_remove(struct idxd_dev *idxd_dev)
 	idxd_wq_del_cdev(wq);
 	drv_disable_wq(wq);
 	wq->type = IDXD_WQT_NONE;
+	destroy_workqueue(wq->wq);
+	wq->wq = NULL;
 	mutex_unlock(&wq->wq_lock);
 }
 
drivers/dma/idxd/idxd.h +1 -0
index b923b90b7299..6e56361ae658 100644
@@ -185,6 +185,7 @@ struct idxd_wq {
 	struct idxd_dev idxd_dev;
 	struct idxd_cdev *idxd_cdev;
 	struct wait_queue_head err_queue;
+	struct workqueue_struct *wq;
 	struct idxd_device *idxd;
 	int id;
 	struct idxd_irq_entry ie;
Signature · 1 line
2.37.1
author
#7 lore raw

PATCH v2 7/16 dmaengine: idxd: create kmem cache for event log fault items

In reply to Fenghua Yu

From: Dave Jiang <[email protected]>

Add a kmem cache per device for allocating event log fault context. The context allows an event log entry to be copied and passed to a software workqueue to be processed. Due to each device can have different sized event log entry depending on device type, it's not possible to have a global kmem cache.

---
 drivers/dma/idxd/idxd.h  | 10 ++++++++++
 drivers/dma/idxd/init.c  |  9 +++++++++
 drivers/dma/idxd/sysfs.c |  1 +
 3 files changed, 20 insertions(+)
drivers/dma/idxd/idxd.h +10 -0
index 6e56361ae658..c5d99c179902 100644
@@ -274,6 +274,15 @@ struct idxd_evl {
 	u16 head;
 };
 
+struct idxd_evl_fault {
+	struct work_struct work;
+	struct idxd_wq *wq;
+	u8 status;
+
+	/* make this last member always */
+	struct __evl_entry entry[];
+};
+
 struct idxd_device {
 	struct idxd_dev idxd_dev;
 	struct idxd_driver_data *data;
@@ -331,6 +340,7 @@ struct idxd_device {
 
 	unsigned long *opcap_bmap;
 	struct idxd_evl *evl;
+	struct kmem_cache *evl_cache;
 
 	struct dentry *dbgfs_dir;
 	struct dentry *dbgfs_evl_file;
drivers/dma/idxd/init.c +9 -0
index d19bc6389221..a7c98fac7a85 100644
@@ -346,6 +346,15 @@ static int idxd_init_evl(struct idxd_device *idxd)
 
 	spin_lock_init(&evl->lock);
 	evl->size = IDXD_EVL_SIZE_MIN;
+
+	idxd->evl_cache = kmem_cache_create(dev_name(idxd_confdev(idxd)),
+					    sizeof(struct idxd_evl_fault) + evl_ent_size(idxd),
+					    0, 0, NULL);
+	if (!idxd->evl_cache) {
+		kfree(evl);
+		return -ENOMEM;
+	}
+
 	idxd->evl = evl;
 	return 0;
 }
drivers/dma/idxd/sysfs.c +1 -0
index 163fdfaa5022..8b9dfa0d2b99 100644
@@ -1718,6 +1718,7 @@ static void idxd_conf_device_release(struct device *dev)
 	kfree(idxd->wqs);
 	kfree(idxd->engines);
 	kfree(idxd->evl);
+	kmem_cache_destroy(idxd->evl_cache);
 	ida_free(&idxd_ida, idxd->id);
 	bitmap_free(idxd->opcap_bmap);
 	kfree(idxd);
Signature · 1 line
2.37.1
author
#8 lore raw

PATCH v2 8/16 iommu: define and export iommu_access_remote_vm()

In reply to Fenghua Yu

Define and export iommu_access_remote_vm() to allow IOMMU related drivers to access user address space by PASID.

The IDXD driver would like to use it to write the user's completion record that the hardware device is not able to write to due to user page fault.

Without the API, it's complex for IDXD driver to copy completion record to a process' fault address for two reasons:
1. access_remote_vm() is not exported and shouldn't be exported for drivers because drivers may easily cause mm reference issue.
2. user frees fault address pages to trigger fault by IDXD device.

The driver has to call iommu_sva_find(), kthread_use_mm(), re-implement majority of access_remote_vm() etc to access remote vm.

This IOMMU specific API hides these details and provides a clean interface for idxd driver and potentially other IOMMU related drivers.

Cc Joerg Roedel <[email protected]>
Cc Will Deacon <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Lorenzo Stoakes <[email protected]>
Cc Christoph Hellwig <[email protected]>

---
v2:
- Define and export iommu_access_remote_vm() for IDXD driver to write completion record to user address space. This change removes patch 8 and 9 in v1 (Alistair Popple)

 drivers/iommu/iommu-sva.c | 35 +++++++++++++++++++++++++++++++++++
 include/linux/iommu.h     |  9 +++++++++
 2 files changed, 44 insertions(+)
drivers/iommu/iommu-sva.c +35 -0
index 24bf9b2b58aa..1d7a0aee58f7 100644
@@ -71,6 +71,41 @@ struct mm_struct *iommu_sva_find(ioasid_t pasid)
 }
 EXPORT_SYMBOL_GPL(iommu_sva_find);
 
+/**
+ * iommu_access_remote_vm - access another process' address space by PASID
+ * @pasid:	Process Address Space ID assigned to the mm
+ * @addr:	start address to access
+ * @buf:	source or destination buffer
+ * @len:	number of bytes to transfer
+ * @gup_flags:	flags modifying lookup behaviour
+ *
+ * Another process' address space is found by PASID. A reference on @mm
+ * is taken and released inside the function.
+ *
+ * Return: number of bytes copied from source to destination.
+ */
+int iommu_access_remote_vm(ioasid_t pasid, unsigned long addr, void *buf,
+			   int len, unsigned int gup_flags)
+{
+	struct mm_struct *mm;
+	int copied;
+
+	mm = iommu_sva_find(pasid);
+	if (IS_ERR_OR_NULL(mm))
+		return 0;
+
+	/*
+	 * A reference on @mm has been held by mmget_not_zero()
+	 * during iommu_sva_find().
+	 */
+	copied = access_remote_vm(mm, addr, buf, len, gup_flags);
+	/* The reference is released. */
+	mmput(mm);
+
+	return copied;
+}
+EXPORT_SYMBOL_GPL(iommu_access_remote_vm);
+
 /**
  * iommu_sva_bind_device() - Bind a process address space to a device
  * @dev: the device
include/linux/iommu.h +9 -0
index 6595454d4f48..414a46a53799 100644
@@ -1177,6 +1177,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev,
 					struct mm_struct *mm);
 void iommu_sva_unbind_device(struct iommu_sva *handle);
 u32 iommu_sva_get_pasid(struct iommu_sva *handle);
+int iommu_access_remote_vm(ioasid_t pasid, unsigned long addr, void *buf,
+			   int len, unsigned int gup_flags);
 #else
 static inline struct iommu_sva *
 iommu_sva_bind_device(struct device *dev, struct mm_struct *mm)
@@ -1192,6 +1194,13 @@ static inline u32 iommu_sva_get_pasid(struct iommu_sva *handle)
 {
 	return IOMMU_PASID_INVALID;
 }
+
+static inline int iommu_access_remote_vm(ioasid_t pasid, unsigned long addr,
+					 void *buf, int len,
+					 unsigned int gup_flags)
+{
+	return 0;
+}
 #endif /* CONFIG_IOMMU_SVA */
 
 #endif /* __LINUX_IOMMU_H */
Signature · 1 line
2.37.1
#9 lore raw

Re: [PATCH v2 08/16] iommu: define and export iommu_access_remote_vm()

Quoted 6 lines · Fenghua Yu

Define and export iommu_access_remote_vm() to allow IOMMU related drivers to access user address space by PASID.

The IDXD driver would like to use it to write the user's completion record that the hardware device is not able to write to due to user page fault.

I don't quite follow here. Isn't I/O page fault already supported?

Best regards,
baolu

author
#10 lore raw

Re: [PATCH v2 08/16] iommu: define and export iommu_access_remote_vm()

Hi, Baolu,

Quoted 9 lines · Baolu Lu
Quoted 6 lines · Fenghua Yu

Define and export iommu_access_remote_vm() to allow IOMMU related drivers to access user address space by PASID.

The IDXD driver would like to use it to write the user's completion record that the hardware device is not able to write to due to user page fault.

I don't quite follow here. Isn't I/O page fault already supported?

The following patch 9 in this series explains in details why IDXD device cannot use page fault to write to user memory: https://lore.kernel.org/dmaengine/[email protected]/

"DSA supports page fault handling through PRS. However, the DMA engine that's processing the descriptor is blocked until the PRS response is received. Other workqueues sharing the engine are also blocked. Page fault handing by the driver with PRS disabled can be used to mitigate the stalling.

With PRS disabled while ATS remain enabled, DSA handles page faults on a completion record by reporting an event in the event log. In this instance, the descriptor is completed and the event log contains the completion record address and the contents of the completion record."

That's why IDXD driver needs this IOMMU's helper iommu_access_remote_vm() to copy the completion record from event log buffer to user space.

Thanks.

-Fenghua

#11 lore raw

Re: [PATCH v2 08/16] iommu: define and export iommu_access_remote_vm()

Quoted 13 lines · Fenghua Yu

Hi, Baolu,

Quoted 9 lines · Baolu Lu
Quoted 6 lines · Fenghua Yu

Define and export iommu_access_remote_vm() to allow IOMMU related drivers to access user address space by PASID.

The IDXD driver would like to use it to write the user's completion record that the hardware device is not able to write to due to user page fault.

I don't quite follow here. Isn't I/O page fault already supported?

Quoted 7 lines

The following patch 9 in this series explains in details why IDXD device cannot use page fault to write to user memory: https://lore.kernel.org/dmaengine/[email protected]/

"DSA supports page fault handling through PRS. However, the DMA engine that's processing the descriptor is blocked until the PRS response is received. Other workqueues sharing the engine are also blocked. Page fault handing by the driver with PRS disabled can be used to mitigate the stalling.

Ah! Get your point now. Thanks for the explanation.

Quoted 11 lines

With PRS disabled while ATS remain enabled, DSA handles page faults on a completion record by reporting an event in the event log. In this instance, the descriptor is completed and the event log contains the completion record address and the contents of the completion record."

That's why IDXD driver needs this IOMMU's helper iommu_access_remote_vm() to copy the completion record from event log buffer to user space.

Thanks.

-Fenghua

Best regards,
baolu

#12 lore raw

Re: [PATCH v2 08/16] iommu: define and export iommu_access_remote_vm()

In reply to Fenghua Yu
Quoted 12 lines · Fenghua Yu
Quoted 2 lines

I don't quite follow here. Isn't I/O page fault already supported?

The following patch 9 in this series explains in details why IDXD device
cannot use page fault to write to user memory: https://lore.kernel.org/dmaengine/[email protected]/

"DSA supports page fault handling through PRS. However, the DMA engine that's processing the descriptor is blocked until the PRS response is received. Other workqueues sharing the engine are also blocked. Page fault handing by the driver with PRS disabled can be used to mitigate the stalling.

Quoted 4 lines

With PRS disabled while ATS remain enabled, DSA handles page faults on a completion record by reporting an event in the event log. In this instance, the descriptor is completed and the event log contains the completion record address and the contents of the completion record."

This seems like a completely broken I/O model, and I don't think Linux should support this model when it requires operations like access_remote_vm.

author
#13 lore raw

Re: [PATCH v2 08/16] iommu: define and export iommu_access_remote_vm()

Hi, Christoph,

Quoted 13 lines · Christoph Hellwig
Quoted 12 lines · Fenghua Yu
Quoted 2 lines

I don't quite follow here. Isn't I/O page fault already supported?

The following patch 9 in this series explains in details why IDXD device
cannot use page fault to write to user memory: https://lore.kernel.org/dmaengine/[email protected]/

"DSA supports page fault handling through PRS. However, the DMA engine that's processing the descriptor is blocked until the PRS response is received. Other workqueues sharing the engine are also blocked. Page fault handing by the driver with PRS disabled can be used to mitigate the stalling.

Quoted 8 lines
Quoted 4 lines

With PRS disabled while ATS remain enabled, DSA handles page faults on a completion record by reporting an event in the event log. In this instance, the descriptor is completed and the event log contains the completion record address and the contents of the completion record."

This seems like a completely broken I/O model, and I don't think Linux should support this model when it requires operations like access_remote_vm.

This patch set have two parts:
1. Basic event log support and PRS disabling knob.
2. Completion record page fault fixup part. The current patch is the major patch in this part. It tries to fix up completion record page fault from user space.

Since the fix up in part 2 is debatable and part 1 can be sent out independently, how about we send out the parts separately?

In the new part 1, simply warn on completion record page fault and don't try to fix it up. Usually a process executes ENQCMD instruction and then keeps polling completion record periodically. So the completion record will be likely backed by page and won't generate page fault. If page fault is really an issue, sysadmin can enable PRS (which is enabled by default anyway) and let PRS to handle page fault.

Then in the next step, we will send out a new part 2 to eliminate completion record page fault. One proposal is to mmap() kernel allocated completion record area. So there won't be completion record page fault and fix up(no access_remote_vm() of course).

Is this OK for you?

Thank you very much for your comment!

-Fenghua

#14 lore raw

Re: [PATCH v2 08/16] iommu: define and export iommu_access_remote_vm()

In reply to Fenghua Yu

Hi Fenghua,

Quoted 56 lines · Fenghua Yu

Define and export iommu_access_remote_vm() to allow IOMMU related drivers to access user address space by PASID.

The IDXD driver would like to use it to write the user's completion record that the hardware device is not able to write to due to user page fault.

Without the API, it's complex for IDXD driver to copy completion record to a process' fault address for two reasons:
1. access_remote_vm() is not exported and shouldn't be exported for drivers because drivers may easily cause mm reference issue.
2. user frees fault address pages to trigger fault by IDXD device.

The driver has to call iommu_sva_find(), kthread_use_mm(), re-implement majority of access_remote_vm() etc to access remote vm.

This IOMMU specific API hides these details and provides a clean interface for idxd driver and potentially other IOMMU related drivers.

Cc Joerg Roedel <[email protected]>
Cc Will Deacon <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Lorenzo Stoakes <[email protected]>
Cc Christoph Hellwig <[email protected]>

---
v2:
- Define and export iommu_access_remote_vm() for IDXD driver to write completion record to user address space. This change removes patch 8 and 9 in v1 (Alistair Popple)

 drivers/iommu/iommu-sva.c | 35 +++++++++++++++++++++++++++++++++++
 include/linux/iommu.h     |  9 +++++++++
 2 files changed, 44 insertions(+)
drivers/iommu/iommu-sva.c +10 -0
index 24bf9b2b58aa..1d7a0aee58f7 100644
@@ -71,6 +71,41 @@ struct mm_struct *iommu_sva_find(ioasid_t pasid)
 }
 EXPORT_SYMBOL_GPL(iommu_sva_find);
 
+/**
+ * iommu_access_remote_vm - access another process' address space by PASID
+ * @pasid:	Process Address Space ID assigned to the mm
+ * @addr:	start address to access
+ * @buf:	source or destination buffer
+ * @len:	number of bytes to transfer
+ * @gup_flags:	flags modifying lookup behaviour
+ *
+ * Another process' address space is found by PASID. A reference on @mm
+ * is taken and released inside the function.

drivers/iommu/iommu-sva.c struct mm_struct *iommu_sva_find(ioasid_t pasid) · line 93

Quoted 10 lines
drivers/iommu/iommu-sva.c +10 -0
+ *
+ * Return: number of bytes copied from source to destination.
+ */
+int iommu_access_remote_vm(ioasid_t pasid, unsigned long addr, void *buf,
+			   int len, unsigned int gup_flags)
+{
+	struct mm_struct *mm;
+	int copied;
+
+	mm = iommu_sva_find(pasid);

The ability to find a mm by PASID is being removed, see
https://lore.kernel.org/linux-iommu/[email protected]/

Thanks,
Jean

Quoted 49 lines
drivers/iommu/iommu-sva.c +15 -0
+	if (IS_ERR_OR_NULL(mm))
+		return 0;
+
+	/*
+	 * A reference on @mm has been held by mmget_not_zero()
+	 * during iommu_sva_find().
+	 */
+	copied = access_remote_vm(mm, addr, buf, len, gup_flags);
+	/* The reference is released. */
+	mmput(mm);
+
+	return copied;
+}
+EXPORT_SYMBOL_GPL(iommu_access_remote_vm);
+
 /**
  * iommu_sva_bind_device() - Bind a process address space to a device
  * @dev: the device
include/linux/iommu.h +9 -0
index 6595454d4f48..414a46a53799 100644
@@ -1177,6 +1177,8 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev,
 					struct mm_struct *mm);
 void iommu_sva_unbind_device(struct iommu_sva *handle);
 u32 iommu_sva_get_pasid(struct iommu_sva *handle);
+int iommu_access_remote_vm(ioasid_t pasid, unsigned long addr, void *buf,
+			   int len, unsigned int gup_flags);
 #else
 static inline struct iommu_sva *
 iommu_sva_bind_device(struct device *dev, struct mm_struct *mm)
@@ -1192,6 +1194,13 @@ static inline u32 iommu_sva_get_pasid(struct iommu_sva *handle)
 {
 	return IOMMU_PASID_INVALID;
 }
+
+static inline int iommu_access_remote_vm(ioasid_t pasid, unsigned long addr,
+					 void *buf, int len,
+					 unsigned int gup_flags)
+{
+	return 0;
+}
 #endif /* CONFIG_IOMMU_SVA */
 
 #endif /* __LINUX_IOMMU_H */
Signature · 1 line
2.37.1
author
#15 lore raw

Re: [PATCH v2 08/16] iommu: define and export iommu_access_remote_vm()

Hi, Jean,

Quoted 64 lines · Jean-Philippe Brucker

Hi Fenghua,

Quoted 61 lines · Fenghua Yu

Define and export iommu_access_remote_vm() to allow IOMMU related drivers to access user address space by PASID.

The IDXD driver would like to use it to write the user's completion record that the hardware device is not able to write to due to user page fault.

Without the API, it's complex for IDXD driver to copy completion record to a process' fault address for two reasons:
1. access_remote_vm() is not exported and shouldn't be exported for drivers because drivers may easily cause mm reference issue.
2. user frees fault address pages to trigger fault by IDXD device.

The driver has to call iommu_sva_find(), kthread_use_mm(), re-implement majority of access_remote_vm() etc to access remote vm.

This IOMMU specific API hides these details and provides a clean interface for idxd driver and potentially other IOMMU related drivers.

Cc Joerg Roedel <[email protected]>
Cc Will Deacon <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Lorenzo Stoakes <[email protected]>
Cc Christoph Hellwig <[email protected]>

---
v2:
- Define and export iommu_access_remote_vm() for IDXD driver to write completion record to user address space. This change removes patch 8 and 9 in v1 (Alistair Popple)

  drivers/iommu/iommu-sva.c | 35 +++++++++++++++++++++++++++++++++++
  include/linux/iommu.h     |  9 +++++++++
  2 files changed, 44 insertions(+)
drivers/iommu/iommu-sva.c +15 -0
index 24bf9b2b58aa..1d7a0aee58f7 100644
@@ -71,6 +71,41 @@ struct mm_struct *iommu_sva_find(ioasid_t pasid)
  }
  EXPORT_SYMBOL_GPL(iommu_sva_find);
 
+/**
+ * iommu_access_remote_vm - access another process' address space by PASID
+ * @pasid:	Process Address Space ID assigned to the mm
+ * @addr:	start address to access
+ * @buf:	source or destination buffer
+ * @len:	number of bytes to transfer
+ * @gup_flags:	flags modifying lookup behaviour
+ *
+ * Another process' address space is found by PASID. A reference on @mm
+ * is taken and released inside the function.
+ *
+ * Return: number of bytes copied from source to destination.
+ */
+int iommu_access_remote_vm(ioasid_t pasid, unsigned long addr, void *buf,
+			   int len, unsigned int gup_flags)
Quoted 9 lines

drivers/iommu/iommu-sva.c struct mm_struct *iommu_sva_find(ioasid_t pasid) · line 93

Quoted 5 lines
drivers/iommu/iommu-sva.c +5 -0
+{
+	struct mm_struct *mm;
+	int copied;
+
+	mm = iommu_sva_find(pasid);

The ability to find a mm by PASID is being removed, see
https://lore.kernel.org/linux-iommu/[email protected]/

Thank you very much for pointing out this.

I talked to Jacob just now. He will keep iommu_sva_find() function
in his next version because this patch is still using the function. He agrees that I can still call iommu_sva_find() in this patch.

-Fenghua

author
#16 lore raw

Re: [PATCH v2 08/16] iommu: define and export iommu_access_remote_vm()

Hi, Jean and Jacob,

Quoted 79 lines · Fenghua Yu

Hi, Jean,

Quoted 65 lines · Jean-Philippe Brucker

Hi Fenghua,

Quoted 62 lines · Fenghua Yu

Define and export iommu_access_remote_vm() to allow IOMMU related drivers to access user address space by PASID.

The IDXD driver would like to use it to write the user's completion record that the hardware device is not able to write to due to user page fault.

Without the API, it's complex for IDXD driver to copy completion record to a process' fault address for two reasons:
1. access_remote_vm() is not exported and shouldn't be exported for drivers because drivers may easily cause mm reference issue.
2. user frees fault address pages to trigger fault by IDXD device.

The driver has to call iommu_sva_find(), kthread_use_mm(), re-implement majority of access_remote_vm() etc to access remote vm.

This IOMMU specific API hides these details and provides a clean interface for idxd driver and potentially other IOMMU related drivers.

Cc Joerg Roedel <[email protected]>
Cc Will Deacon <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Alistair Popple <[email protected]>
Cc Lorenzo Stoakes <[email protected]>
Cc Christoph Hellwig <[email protected]>

---
v2:
- Define and export iommu_access_remote_vm() for IDXD driver to write completion record to user address space. This change removes patch 8 and 9 in v1 (Alistair Popple)

  drivers/iommu/iommu-sva.c | 35 +++++++++++++++++++++++++++++++++++ include/linux/iommu.h     |  9 +++++++++

  2 files changed, 44 insertions(+)
drivers/iommu/iommu-sva.c +2 -0
index 24bf9b2b58aa..1d7a0aee58f7 100644
@@ -71,6 +71,41 @@ struct mm_struct *iommu_sva_find(ioasid_t pasid)
   }
   EXPORT_SYMBOL_GPL(iommu_sva_find);
+/**
+ * iommu_access_remote_vm - access another process' address space by 

PASID
+ * @pasid:    Process Address Space ID assigned to the mm
+ * @addr:    start address to access
+ * @buf:    source or destination buffer
+ * @len:    number of bytes to transfer
+ * @gup_flags:    flags modifying lookup behaviour
+ *
+ * Another process' address space is found by PASID. A reference on @mm
+ * is taken and released inside the function.
+ *
+ * Return: number of bytes copied from source to destination.
+ */
+int iommu_access_remote_vm(ioasid_t pasid, unsigned long addr, void *buf,
+               int len, unsigned int gup_flags)

Quoted 10 lines

drivers/iommu/iommu-sva.c struct mm_struct *iommu_sva_find(ioasid_t pasid) · line 93

Quoted 5 lines
drivers/iommu/iommu-sva.c +5 -0
+{
+    struct mm_struct *mm;
+    int copied;
+
+    mm = iommu_sva_find(pasid);

The ability to find a mm by PASID is being removed, see
https://lore.kernel.org/linux-iommu/[email protected]/

Quoted 4 lines

Thank you very much for pointing out this.

I talked to Jacob just now. He will keep iommu_sva_find() function
in his next version because this patch is still using the function. He agrees that I can still call iommu_sva_find() in this patch.

Further comment from Jason confirms that iommu_sva_find() will be removed (https://lore.kernel.org/lkml/ZAjSsm4%[email protected]/).

So cannot call iommu_sva_find() any more. Will maintain mm and find mm from PASID inside IDXD driver. And will implement accessing the remote mm inside IDXD driver although the implementation will have duplicate code as access_remote_vm().

Next version will only change IDXD driver code. There won't be IOMMU code change.

Thanks.

-Fenghua

author
#17 lore raw

PATCH v2 9/16 dmaengine: idxd: process user page faults for completion record

In reply to Fenghua Yu

From: Dave Jiang <[email protected]>

DSA supports page fault handling through PRS. However, the DMA engine that's processing the descriptor is blocked until the PRS response is received. Other workqueues sharing the engine are also blocked. Page fault handing by the driver with PRS disabled can be used to mitigate the stalling.

With PRS disabled while ATS remain enabled, DSA handles page faults on a completion record by reporting an event in the event log. In this instance, the descriptor is completed and the event log contains the completion record address and the contents of the completion record. Add support to the event log handling code to fault in the completion record and copy the content of the completion record to user memory.

A bitmap is introduced to keep track of discarded event log entries. When the user process initiates ->release() of the char device, it no longer is interested in any remaining event log entries tied to the relevant wq and PASID. The driver will mark the event log entry index in the bitmap. Upon encountering the entries during processing, the event log handler will just clear the bitmap bit and skip the entry rather than attempt to process the event log entry.

---
v2:
- Call iommu_access_remote_vm() to copy completion record to user.

 drivers/dma/Kconfig       |  1 +
 drivers/dma/idxd/cdev.c   | 34 ++++++++++++++++-
 drivers/dma/idxd/device.c | 22 ++++++++++-
 drivers/dma/idxd/idxd.h   |  2 +
 drivers/dma/idxd/init.c   |  2 +
 drivers/dma/idxd/irq.c    | 79 ++++++++++++++++++++++++++++++++++++---
 include/uapi/linux/idxd.h |  1 +
 7 files changed, 133 insertions(+), 8 deletions(-)
drivers/dma/Kconfig +1 -0
index fb7073fc034f..c8a2d255930e 100644
@@ -297,6 +297,7 @@ config INTEL_IDXD
 	depends on PCI_PASID
 	depends on SBITMAP
 	select DMA_ENGINE
+	select IOMMU_SVA
 	help
 	  Enable support for the Intel(R) data accelerators present
 	  in Intel Xeon CPU.
drivers/dma/idxd/cdev.c +33 -1
index cbe29e1a6a44..51a5b8ab160e 100644
@@ -136,6 +136,35 @@ static int idxd_cdev_open(struct inode *inode, struct file *filp)
 	return rc;
 }
 
+static void idxd_cdev_evl_drain_pasid(struct idxd_wq *wq, u32 pasid)
+{
+	struct idxd_device *idxd = wq->idxd;
+	struct idxd_evl *evl = idxd->evl;
+	union evl_status_reg status;
+	u16 h, t, size;
+	int ent_size = evl_ent_size(idxd);
+	struct __evl_entry *entry_head;
+
+	if (!evl)
+		return;
+
+	spin_lock(&evl->lock);
+	status.bits = ioread64(idxd->reg_base + IDXD_EVLSTATUS_OFFSET);
+	t = status.tail;
+	h = evl->head;
+	size = evl->size;
+
+	while (h != t) {
+		entry_head = (struct __evl_entry *)(evl->log + (h * ent_size));
+		if (entry_head->pasid == pasid && entry_head->wq_idx == wq->id)
+			set_bit(h, evl->bmap);
+		h = (h + 1) % size;
+	}
+	spin_unlock(&evl->lock);
+
+	drain_workqueue(wq->wq);
+}
+
 static int idxd_cdev_release(struct inode *node, struct file *filep)
 {
 	struct idxd_user_context *ctx = filep->private_data;
@@ -161,8 +190,11 @@ static int idxd_cdev_release(struct inode *node, struct file *filep)
 		}
 	}
 
-	if (ctx->sva)
+	if (ctx->sva) {
+		idxd_cdev_evl_drain_pasid(wq, ctx->pasid);
 		iommu_sva_unbind_device(ctx->sva);
+	}
+
 	kfree(ctx);
 	mutex_lock(&wq->wq_lock);
 	idxd_wq_put(wq);
drivers/dma/idxd/device.c +20 -2
index 230fe9bb56ae..fd97b2b58734 100644
@@ -762,18 +762,29 @@ static int idxd_device_evl_setup(struct idxd_device *idxd)
 	dma_addr_t dma_addr;
 	int size;
 	struct idxd_evl *evl = idxd->evl;
+	unsigned long *bmap;
+	int rc;
 
 	if (!evl)
 		return 0;
 
 	size = evl_size(idxd);
+
+	bmap = bitmap_zalloc(size, GFP_KERNEL);
+	if (!bmap) {
+		rc = -ENOMEM;
+		goto err_bmap;
+	}
+
 	/*
 	 * Address needs to be page aligned. However, dma_alloc_coherent() provides
 	 * at minimal page size aligned address. No manual alignment required.
 	 */
 	addr = dma_alloc_coherent(dev, size, &dma_addr, GFP_KERNEL);
-	if (!addr)
-		return -ENOMEM;
+	if (!addr) {
+		rc = -ENOMEM;
+		goto err_alloc;
+	}
 
 	memset(addr, 0, size);
 
@@ -781,6 +792,7 @@ static int idxd_device_evl_setup(struct idxd_device *idxd)
 	evl->log = addr;
 	evl->dma = dma_addr;
 	evl->log_size = size;
+	evl->bmap = bmap;
 
 	memset(&evlcfg, 0, sizeof(evlcfg));
 	evlcfg.bits[0] = dma_addr & GENMASK(63, 12);
@@ -799,6 +811,11 @@ static int idxd_device_evl_setup(struct idxd_device *idxd)
 
 	spin_unlock(&evl->lock);
 	return 0;
+
+err_alloc:
+	bitmap_free(bmap);
+err_bmap:
+	return rc;
 }
 
 static void idxd_device_evl_free(struct idxd_device *idxd)
@@ -824,6 +841,7 @@ static void idxd_device_evl_free(struct idxd_device *idxd)
 	iowrite64(0, idxd->reg_base + IDXD_EVLCFG_OFFSET + 8);
 
 	dma_free_coherent(dev, evl->log_size, evl->log, evl->dma);
+	bitmap_free(evl->bmap);
 	evl->log = NULL;
 	evl->size = IDXD_EVL_SIZE_MIN;
 	spin_unlock(&evl->lock);
drivers/dma/idxd/idxd.h +2 -0
index c5d99c179902..48cdfd5ee44f 100644
@@ -260,6 +260,7 @@ struct idxd_driver_data {
 	struct device_type *dev_type;
 	int compl_size;
 	int align;
+	int evl_cr_off;
 };
 
 struct idxd_evl {
@@ -272,6 +273,7 @@ struct idxd_evl {
 	/* The number of entries in the event log. */
 	u16 size;
 	u16 head;
+	unsigned long *bmap;
 };
 
 struct idxd_evl_fault {
drivers/dma/idxd/init.c +2 -0
index a7c98fac7a85..19324fbc238c 100644
@@ -47,6 +47,7 @@ static struct idxd_driver_data idxd_driver_data[] = {
 		.compl_size = sizeof(struct dsa_completion_record),
 		.align = 32,
 		.dev_type = &dsa_device_type,
+		.evl_cr_off = offsetof(struct dsa_evl_entry, cr),
 	},
 	[IDXD_TYPE_IAX] = {
 		.name_prefix = "iax",
@@ -54,6 +55,7 @@ static struct idxd_driver_data idxd_driver_data[] = {
 		.compl_size = sizeof(struct iax_completion_record),
 		.align = 64,
 		.dev_type = &iax_device_type,
+		.evl_cr_off = offsetof(struct iax_evl_entry, cr),
 	},
 };
 
drivers/dma/idxd/irq.c +74 -5

101 lines not on this page · raw message

include/uapi/linux/idxd.h +1 -0
index e86199d09a91..4b584d5afd87 100644
@@ -135,6 +135,7 @@ enum dsa_completion_status {
 	DSA_COMP_HW_ERR1,
 	DSA_COMP_HW_ERR_DRB,
 	DSA_COMP_TRANSLATION_FAIL,
+	DSA_COMP_DRAIN_EVL = 0x26,
 };
 
 enum iax_completion_status {
Signature · 1 line
2.37.1
author
#18 lore raw

PATCH v2 10/16 dmaengine: idxd: add descs_completed field for completion record

In reply to Fenghua Yu

From: Dave Jiang <[email protected]>

The descs_completed field for a completion record is part of a batch descriptor completion record. It takes the same location as bytes_completed in a normal descriptor field. Add to expose to user.

---
 include/uapi/linux/idxd.h | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
include/uapi/linux/idxd.h +4 -1
index 4b584d5afd87..76ad71bf751e 100644
@@ -314,7 +314,10 @@ struct dsa_completion_record {
 	};
 	uint8_t			fault_info;
 	uint8_t			rsvd;
-	uint32_t		bytes_completed;
+	union {
+		uint32_t		bytes_completed;
+		uint32_t		descs_completed;
+	};
 	uint64_t		fault_addr;
 	union {
 		/* common record */
Signature · 1 line
2.37.1
author
#19 lore raw

PATCH v2 11/16 dmaengine: idxd: process batch descriptor completion record faults

In reply to Fenghua Yu

From: Dave Jiang <[email protected]>

Add event log processing for faulting of user batch descriptor completion record.

When encountering an event log entry for a page fault on a completion record, the driver is expected to do the following:
1. If the "first error in batch" bit in event log entry error info is set, discard any previously recorded errors associated with the "batch identifier".
2. Fix the page fault according to the fault address in the event log. If successful, write the completion record to the fault address in user space.
3. If an error is encountered while writing the completion record and it is associated to a descriptor in the batch, the driver associates the error with the batch identifier of the event log entry and tracks it until the event log entry for the corresponding batch desc is encountered.

While processing an event log entry for a batch descriptor with error indicating that one or more descs in the batch had event log entries, the driver will do the following before writing the batch completion record:
1. If the status field of the completion record is 0x1, the driver will change it to error code 0x5 (one or more operations in batch completed with status not successful) and changes the result field to 1.
2. If the status is error code 0x6 (page fault on batch descriptor list address), change the result field to 1.
3. If status is any other value, the completion record is not changed.
4. Clear the recorded error in preparation for next batch with same batch identifier.

The result field is for user software to determine whether to set the "Batch Error" flag bit in the descriptor for continuation of partial batch descriptor completion. See DSA spec 2.0 for additional information.

If no error has been recorded for the batch, the batch completion record is written to user space as is.

---
v2:
- Call iommu_access_remote_vm() to copy completion record to user.

 drivers/dma/idxd/idxd.h      |  3 ++
 drivers/dma/idxd/init.c      |  4 ++
 drivers/dma/idxd/irq.c       | 74 ++++++++++++++++++++++++++++--------
 drivers/dma/idxd/registers.h |  4 +-
 include/uapi/linux/idxd.h    |  1 +
 5 files changed, 70 insertions(+), 16 deletions(-)
drivers/dma/idxd/idxd.h +3 -0
index 48cdfd5ee44f..8f3b0fbd04ae 100644
@@ -261,6 +261,8 @@ struct idxd_driver_data {
 	int compl_size;
 	int align;
 	int evl_cr_off;
+	int cr_status_off;
+	int cr_result_off;
 };
 
 struct idxd_evl {
@@ -274,6 +276,7 @@ struct idxd_evl {
 	u16 size;
 	u16 head;
 	unsigned long *bmap;
+	bool batch_fail[IDXD_MAX_BATCH_IDENT];
 };
 
 struct idxd_evl_fault {
drivers/dma/idxd/init.c +4 -0
index 19324fbc238c..73fb9c74ed20 100644
@@ -48,6 +48,8 @@ static struct idxd_driver_data idxd_driver_data[] = {
 		.align = 32,
 		.dev_type = &dsa_device_type,
 		.evl_cr_off = offsetof(struct dsa_evl_entry, cr),
+		.cr_status_off = offsetof(struct dsa_completion_record, status),
+		.cr_result_off = offsetof(struct dsa_completion_record, result),
 	},
 	[IDXD_TYPE_IAX] = {
 		.name_prefix = "iax",
@@ -56,6 +58,8 @@ static struct idxd_driver_data idxd_driver_data[] = {
 		.align = 64,
 		.dev_type = &iax_device_type,
 		.evl_cr_off = offsetof(struct iax_evl_entry, cr),
+		.cr_status_off = offsetof(struct iax_completion_record, status),
+		.cr_result_off = offsetof(struct iax_completion_record, error_code),
 	},
 };
 
drivers/dma/idxd/irq.c +59 -15

94 lines not on this page · raw message

drivers/dma/idxd/registers.h +3 -1
index 148db94f9373..9f3959d001b6 100644
@@ -35,7 +35,7 @@ union gen_cap_reg {
 		u64 drain_readback:1;
 		u64 rsvd2:3;
 		u64 evl_support:2;
-		u64 rsvd4:1;
+		u64 batch_continuation:1;
 		u64 max_xfer_shift:5;
 		u64 max_batch_shift:4;
 		u64 max_ims_mult:6;
@@ -577,6 +577,8 @@ union evl_status_reg {
 	u64 bits;
 } __packed;
 
+#define IDXD_MAX_BATCH_IDENT	256
+
 struct __evl_entry {
 	u64 rsvd:2;
 	u64 desc_valid:1;
include/uapi/linux/idxd.h +1 -0
index 76ad71bf751e..606b52e88ce3 100644
@@ -136,6 +136,7 @@ enum dsa_completion_status {
 	DSA_COMP_HW_ERR_DRB,
 	DSA_COMP_TRANSLATION_FAIL,
 	DSA_COMP_DRAIN_EVL = 0x26,
+	DSA_COMP_BATCH_EVL_ERR,
 };
 
 enum iax_completion_status {
Signature · 1 line
2.37.1
author
#20 lore raw

PATCH v2 12/16 dmaengine: idxd: add per file user counters for completion record faults

In reply to Fenghua Yu

From: Dave Jiang <[email protected]>

Add counters per opened file for the char device in order to keep track how many completion record faults occurred and how many of those faults failed the writeback by the driver after attempt to fault in the page. An xarray is added to associate the PASID with the struct idxd_user_context so the counters can be managed.

---
 drivers/dma/idxd/cdev.c  | 50 +++++++++++++++++++++++++++++++++++++---
 drivers/dma/idxd/idxd.h  | 11 +++++++++
 drivers/dma/idxd/init.c  |  2 ++
 drivers/dma/idxd/irq.c   |  4 ++++
 drivers/dma/idxd/sysfs.c |  1 +
 5 files changed, 65 insertions(+), 3 deletions(-)
drivers/dma/idxd/cdev.c +47 -3
index 51a5b8ab160e..3ce134afa867 100644
@@ -11,6 +11,7 @@
 #include <linux/fs.h>
 #include <linux/poll.h>
 #include <linux/iommu.h>
+#include <linux/xarray.h>
 #include <uapi/linux/idxd.h>
 #include "registers.h"
 #include "idxd.h"
@@ -36,6 +37,7 @@ struct idxd_user_context {
 	unsigned int pasid;
 	unsigned int flags;
 	struct iommu_sva *sva;
+	u64 counters[COUNTER_MAX];
 };
 
 static void idxd_cdev_dev_release(struct device *dev)
@@ -68,6 +70,36 @@ static inline struct idxd_wq *inode_wq(struct inode *inode)
 	return idxd_cdev->wq;
 }
 
+void idxd_user_counter_increment(struct idxd_wq *wq, u32 pasid, int index)
+{
+	struct idxd_user_context *ctx;
+
+	if (index >= COUNTER_MAX)
+		return;
+
+	mutex_lock(&wq->uc_lock);
+	ctx = xa_load(&wq->upasid_xa, pasid);
+	if (!ctx) {
+		mutex_unlock(&wq->uc_lock);
+		return;
+	}
+	ctx->counters[index]++;
+	mutex_unlock(&wq->uc_lock);
+}
+
+static void idxd_xa_pasid_remove(struct idxd_user_context *ctx)
+{
+	struct idxd_wq *wq = ctx->wq;
+	void *ptr;
+
+	mutex_lock(&wq->uc_lock);
+	ptr = xa_cmpxchg(&wq->upasid_xa, ctx->pasid, ctx, NULL, GFP_KERNEL);
+	if (ptr != (void *)ctx)
+		dev_warn(&wq->idxd->pdev->dev, "xarray cmpxchg failed for pasid %u\n",
+			 ctx->pasid);
+	mutex_unlock(&wq->uc_lock);
+}
+
 static int idxd_cdev_open(struct inode *inode, struct file *filp)
 {
 	struct idxd_user_context *ctx;
@@ -108,20 +140,25 @@ static int idxd_cdev_open(struct inode *inode, struct file *filp)
 
 		pasid = iommu_sva_get_pasid(sva);
 		if (pasid == IOMMU_PASID_INVALID) {
-			iommu_sva_unbind_device(sva);
 			rc = -EINVAL;
-			goto failed;
+			goto failed_get_pasid;
 		}
 
 		ctx->sva = sva;
 		ctx->pasid = pasid;
 
+		mutex_lock(&wq->uc_lock);
+		rc = xa_insert(&wq->upasid_xa, pasid, ctx, GFP_KERNEL);
+		mutex_unlock(&wq->uc_lock);
+		if (rc < 0)
+			dev_warn(dev, "PASID entry already exist in xarray.\n");
+
 		if (wq_dedicated(wq)) {
 			rc = idxd_wq_set_pasid(wq, pasid);
 			if (rc < 0) {
 				iommu_sva_unbind_device(sva);
 				dev_err(dev, "wq set pasid failed: %d\n", rc);
-				goto failed;
+				goto failed_set_pasid;
 			}
 		}
 	}
@@ -130,6 +167,12 @@ static int idxd_cdev_open(struct inode *inode, struct file *filp)
 	mutex_unlock(&wq->wq_lock);
 	return 0;
 
+ failed_set_pasid:
+	if (device_user_pasid_enabled(idxd))
+		idxd_xa_pasid_remove(ctx);
+ failed_get_pasid:
+	if (device_user_pasid_enabled(idxd))
+		iommu_sva_unbind_device(sva);
  failed:
 	mutex_unlock(&wq->wq_lock);
 	kfree(ctx);
@@ -193,6 +236,7 @@ static int idxd_cdev_release(struct inode *node, struct file *filep)
 	if (ctx->sva) {
 		idxd_cdev_evl_drain_pasid(wq, ctx->pasid);
 		iommu_sva_unbind_device(ctx->sva);
+		idxd_xa_pasid_remove(ctx);
 	}
 
 	kfree(ctx);
drivers/dma/idxd/idxd.h +11 -0
index 8f3b0fbd04ae..f92db20015fb 100644
@@ -127,6 +127,12 @@ struct idxd_pmu {
 
 #define IDXD_MAX_PRIORITY	0xf
 
+enum {
+	COUNTER_FAULTS = 0,
+	COUNTER_FAULT_FAILS,
+	COUNTER_MAX
+};
+
 enum idxd_wq_state {
 	IDXD_WQ_DISABLED = 0,
 	IDXD_WQ_ENABLED,
@@ -215,6 +221,10 @@ struct idxd_wq {
 	char name[WQ_NAME_SIZE + 1];
 	u64 max_xfer_bytes;
 	u32 max_batch_size;
+
+	/* Lock to protect upasid_xa access. */
+	struct mutex uc_lock;
+	struct xarray upasid_xa;
 };
 
 struct idxd_engine {
@@ -707,6 +717,7 @@ void idxd_cdev_remove(void);
 int idxd_cdev_get_major(struct idxd_device *idxd);
 int idxd_wq_add_cdev(struct idxd_wq *wq);
 void idxd_wq_del_cdev(struct idxd_wq *wq);
+void idxd_user_counter_increment(struct idxd_wq *wq, u32 pasid, int index);
 
 /* perfmon */
 #if IS_ENABLED(CONFIG_INTEL_IDXD_PERFMON)
drivers/dma/idxd/init.c +2 -0
index 73fb9c74ed20..9b3e7f0770d1 100644
@@ -206,6 +206,8 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
 			}
 			bitmap_copy(wq->opcap_bmap, idxd->opcap_bmap, IDXD_MAX_OPCAP_BITS);
 		}
+		mutex_init(&wq->uc_lock);
+		xa_init(&wq->upasid_xa);
 		idxd->wqs[i] = wq;
 	}
 
drivers/dma/idxd/irq.c +4 -0
index 894a73e56cb6..69e0b8e1b3cf 100644
@@ -241,6 +241,7 @@ static void idxd_evl_fault_work(struct work_struct *work)
 			evl->batch_fail[entry_head->batch_id] = false;
 
 		copy_size = cr_size;
+		idxd_user_counter_increment(wq, entry_head->pasid, COUNTER_FAULTS);
 		break;
 	case DSA_COMP_BATCH_EVL_ERR:
 		bf = &evl->batch_fail[entry_head->batch_id];
@@ -252,6 +253,7 @@ static void idxd_evl_fault_work(struct work_struct *work)
 			*result = 1;
 			*bf = false;
 		}
+		idxd_user_counter_increment(wq, entry_head->pasid, COUNTER_FAULTS);
 		break;
 	case DSA_COMP_DRAIN_EVL:
 		copy_size = cr_size;
@@ -275,6 +277,7 @@ static void idxd_evl_fault_work(struct work_struct *work)
 	switch (fault->status) {
 	case DSA_COMP_CRA_XLAT:
 		if (copied != copy_size) {
+			idxd_user_counter_increment(wq, entry_head->pasid, COUNTER_FAULT_FAILS);
 			dev_err(dev, "Failed to write to completion record: (%d:%d)\n",
 				copy_size, copied);
 			if (entry_head->batch)
@@ -283,6 +286,7 @@ static void idxd_evl_fault_work(struct work_struct *work)
 		break;
 	case DSA_COMP_BATCH_EVL_ERR:
 		if (copied != copy_size) {
+			idxd_user_counter_increment(wq, entry_head->pasid, COUNTER_FAULT_FAILS);
 			dev_err(dev, "Failed to write to batch completion record: (%d:%d)\n",
 				copy_size, copied);
 		}
drivers/dma/idxd/sysfs.c +1 -0
index 8b9dfa0d2b99..465d2e7627e4 100644
@@ -1292,6 +1292,7 @@ static void idxd_conf_wq_release(struct device *dev)
 
 	bitmap_free(wq->opcap_bmap);
 	kfree(wq->wqcfg);
+	xa_destroy(&wq->upasid_xa);
 	kfree(wq);
 }
 
Signature · 1 line
2.37.1
author
#21 lore raw

PATCH v2 13/16 dmaengine: idxd: add a device to represent the file opened

In reply to Fenghua Yu

From: Dave Jiang <[email protected]>

Embed a struct device for the user file context in order to export sysfs attributes related with the opened file. Tie the lifetime of the file context to the device. The sysfs entry will be added under the char device.

---
 drivers/dma/idxd/cdev.c | 122 +++++++++++++++++++++++++++++++---------
 drivers/dma/idxd/idxd.h |   2 +
 2 files changed, 97 insertions(+), 27 deletions(-)
drivers/dma/idxd/cdev.c +95 -27
index 3ce134afa867..2ff9a680ea5e 100644
@@ -22,6 +22,13 @@ struct idxd_cdev_context {
 	struct ida minor_ida;
 };
 
+/*
+ * Since user file names are global in DSA devices, define their ida's as
+ * global to avoid conflict file names.
+ */
+static DEFINE_IDA(file_ida);
+static DEFINE_MUTEX(ida_lock);
+
 /*
  * ictx is an array based off of accelerator types. enum idxd_type
  * is used as index
@@ -37,7 +44,60 @@ struct idxd_user_context {
 	unsigned int pasid;
 	unsigned int flags;
 	struct iommu_sva *sva;
+	struct idxd_dev idxd_dev;
 	u64 counters[COUNTER_MAX];
+	int id;
+};
+
+static void idxd_cdev_evl_drain_pasid(struct idxd_wq *wq, u32 pasid);
+static void idxd_xa_pasid_remove(struct idxd_user_context *ctx);
+
+static inline struct idxd_user_context *dev_to_uctx(struct device *dev)
+{
+	struct idxd_dev *idxd_dev = confdev_to_idxd_dev(dev);
+
+	return container_of(idxd_dev, struct idxd_user_context, idxd_dev);
+}
+
+static void idxd_file_dev_release(struct device *dev)
+{
+	struct idxd_user_context *ctx = dev_to_uctx(dev);
+	struct idxd_wq *wq = ctx->wq;
+	struct idxd_device *idxd = wq->idxd;
+	int rc;
+
+	mutex_lock(&ida_lock);
+	ida_free(&file_ida, ctx->id);
+	mutex_unlock(&ida_lock);
+
+	/* Wait for in-flight operations to complete. */
+	if (wq_shared(wq)) {
+		idxd_device_drain_pasid(idxd, ctx->pasid);
+	} else {
+		if (device_user_pasid_enabled(idxd)) {
+			/* The wq disable in the disable pasid function will drain the wq */
+			rc = idxd_wq_disable_pasid(wq);
+			if (rc < 0)
+				dev_err(dev, "wq disable pasid failed.\n");
+		} else {
+			idxd_wq_drain(wq);
+		}
+	}
+
+	if (ctx->sva) {
+		idxd_cdev_evl_drain_pasid(wq, ctx->pasid);
+		iommu_sva_unbind_device(ctx->sva);
+		idxd_xa_pasid_remove(ctx);
+	}
+	kfree(ctx);
+	mutex_lock(&wq->wq_lock);
+	idxd_wq_put(wq);
+	mutex_unlock(&wq->wq_lock);
+}
+
+static struct device_type idxd_cdev_file_type = {
+	.name = "idxd_file",
+	.release = idxd_file_dev_release,
 };
 
 static void idxd_cdev_dev_release(struct device *dev)
@@ -105,10 +165,11 @@ static int idxd_cdev_open(struct inode *inode, struct file *filp)
 	struct idxd_user_context *ctx;
 	struct idxd_device *idxd;
 	struct idxd_wq *wq;
-	struct device *dev;
+	struct device *dev, *fdev;
 	int rc = 0;
 	struct iommu_sva *sva;
 	unsigned int pasid;
+	struct idxd_cdev *idxd_cdev;
 
 	wq = inode_wq(inode);
 	idxd = wq->idxd;
@@ -163,10 +224,41 @@ static int idxd_cdev_open(struct inode *inode, struct file *filp)
 		}
 	}
 
+	idxd_cdev = wq->idxd_cdev;
+	mutex_lock(&ida_lock);
+	ctx->id = ida_alloc(&file_ida, GFP_KERNEL);
+	mutex_unlock(&ida_lock);
+	if (ctx->id < 0) {
+		dev_warn(dev, "ida alloc failure\n");
+		goto failed_ida;
+	}
+	ctx->idxd_dev.type  = IDXD_DEV_CDEV_FILE;
+	fdev = user_ctx_dev(ctx);
+	device_initialize(fdev);
+	fdev->parent = cdev_dev(idxd_cdev);
+	fdev->bus = &dsa_bus_type;
+	fdev->type = &idxd_cdev_file_type;
+
+	rc = dev_set_name(fdev, "file%d", ctx->id);
+	if (rc < 0) {
+		dev_warn(dev, "set name failure\n");
+		goto failed_dev_name;
+	}
+
+	rc = device_add(fdev);
+	if (rc < 0) {
+		dev_warn(dev, "file device add failure\n");
+		goto failed_dev_add;
+	}
+
 	idxd_wq_get(wq);
 	mutex_unlock(&wq->wq_lock);
 	return 0;
 
+ failed_dev_add:
+ failed_dev_name:
+	put_device(fdev);
+ failed_ida:
  failed_set_pasid:
 	if (device_user_pasid_enabled(idxd))
 		idxd_xa_pasid_remove(ctx);
@@ -214,35 +306,10 @@ static int idxd_cdev_release(struct inode *node, struct file *filep)
 	struct idxd_wq *wq = ctx->wq;
 	struct idxd_device *idxd = wq->idxd;
 	struct device *dev = &idxd->pdev->dev;
-	int rc;
 
 	dev_dbg(dev, "%s called\n", __func__);
 	filep->private_data = NULL;
-
-	/* Wait for in-flight operations to complete. */
-	if (wq_shared(wq)) {
-		idxd_device_drain_pasid(idxd, ctx->pasid);
-	} else {
-		if (device_user_pasid_enabled(idxd)) {
-			/* The wq disable in the disable pasid function will drain the wq */
-			rc = idxd_wq_disable_pasid(wq);
-			if (rc < 0)
-				dev_err(dev, "wq disable pasid failed.\n");
-		} else {
-			idxd_wq_drain(wq);
-		}
-	}
-
-	if (ctx->sva) {
-		idxd_cdev_evl_drain_pasid(wq, ctx->pasid);
-		iommu_sva_unbind_device(ctx->sva);
-		idxd_xa_pasid_remove(ctx);
-	}
-
-	kfree(ctx);
-	mutex_lock(&wq->wq_lock);
-	idxd_wq_put(wq);
-	mutex_unlock(&wq->wq_lock);
+	device_unregister(user_ctx_dev(ctx));
 	return 0;
 }
 
@@ -373,6 +440,7 @@ void idxd_wq_del_cdev(struct idxd_wq *wq)
 	struct idxd_cdev *idxd_cdev;
 
 	idxd_cdev = wq->idxd_cdev;
+	ida_destroy(&file_ida);
 	wq->idxd_cdev = NULL;
 	cdev_device_del(&idxd_cdev->cdev, cdev_dev(idxd_cdev));
 	put_device(cdev_dev(idxd_cdev));
drivers/dma/idxd/idxd.h +2 -0
index f92db20015fb..b5d7ef611bae 100644
@@ -32,6 +32,7 @@ enum idxd_dev_type {
 	IDXD_DEV_GROUP,
 	IDXD_DEV_ENGINE,
 	IDXD_DEV_CDEV,
+	IDXD_DEV_CDEV_FILE,
 	IDXD_DEV_MAX_TYPE,
 };
 
@@ -405,6 +406,7 @@ enum idxd_completion_status {
 #define engine_confdev(engine) &engine->idxd_dev.conf_dev
 #define group_confdev(group) &group->idxd_dev.conf_dev
 #define cdev_dev(cdev) &cdev->idxd_dev.conf_dev
+#define user_ctx_dev(ctx) (&(ctx)->idxd_dev.conf_dev)
 
 #define confdev_to_idxd_dev(dev) container_of(dev, struct idxd_dev, conf_dev)
 #define idxd_dev_to_idxd(idxd_dev) container_of(idxd_dev, struct idxd_device, idxd_dev)
Signature · 1 line
2.37.1
author
#22 lore raw

PATCH v2 14/16 dmaengine: idxd: expose fault counters to sysfs

In reply to Fenghua Yu

From: Dave Jiang <[email protected]>

Expose cr_faults and cr_fault_failures counters to the user space. This allows a user app to keep track of how many fault the application is causing with the completion record (CR) and also the number of failures of the CR writeback. Having a high number of cr_fault_failures is bad as the app is submitting descriptors with the CR addresses that are bad. User monitoring daemon may want to consider killing the application as it may be malicious and attempting to flood the device event log.

---
 .../ABI/stable/sysfs-driver-dma-idxd          | 17 +++++++
 drivers/dma/idxd/cdev.c                       | 46 +++++++++++++++++++
 2 files changed, 63 insertions(+)
Documentation/ABI/stable/sysfs-driver-dma-idxd +17 -0
index e01916611452..73ab86196a41 100644
@@ -318,3 +318,20 @@ Description:	Allows control of the number of batch descriptors that can be
 		1 (1/2 of max value), 2 (1/4 of the max value), and 3 (1/8 of
 		the max value). It's visible only on platforms that support
 		the capability.
+
+What:		/sys/bus/dsa/devices/wq<m>.<n>/dsa<x>\!wq<m>.<n>/file<y>/cr_faults
+Date:		Sept 14, 2022
+KernelVersion:	6.4.0
+Contact:	[email protected]
+Description:	Show the number of Completion Record (CR) faults this application
+		has caused.
+
+What:		/sys/bus/dsa/devices/wq<m>.<n>/dsa<x>\!wq<m>.<n>/file<y>/cr_fault_failures
+Date:		Sept 14, 2022
+KernelVersion:	6.4.0
+Contact:	[email protected]
+Description:	Show the number of Completion Record (CR) faults failures that this
+		application has caused. The failure counter is incremented when the
+		driver cannot fault in the address for the CR. Typically this is caused
+		by a bad address programmed in the submitted descriptor or a malicious
+		submitter is using bad CR address on purpose.
drivers/dma/idxd/cdev.c +46 -0

58 lines not on this page · raw message

Signature · 1 line
2.37.1
author
#23 lore raw

PATCH v2 15/16 dmaengine: idxd: add pid to exported sysfs attribute for opened file

In reply to Fenghua Yu

From: Dave Jiang <[email protected]>

Provide the pid of the application for the opened file. This allows the monitor daemon to easily correlate which app opened the file and easily kill the app by pid if that is desired action.

---
 Documentation/ABI/stable/sysfs-driver-dma-idxd |  8 ++++++++
 drivers/dma/idxd/cdev.c                        | 11 +++++++++++
 2 files changed, 19 insertions(+)
Documentation/ABI/stable/sysfs-driver-dma-idxd +8 -0
index 73ab86196a41..5d0df57f5298 100644
@@ -335,3 +335,11 @@ Description:	Show the number of Completion Record (CR) faults failures that this
 		driver cannot fault in the address for the CR. Typically this is caused
 		by a bad address programmed in the submitted descriptor or a malicious
 		submitter is using bad CR address on purpose.
+
+What:		/sys/bus/dsa/devices/wq<m>.<n>/dsa<x>\!wq<m>.<n>/file<y>/pid
+Date:		Sept 14, 2022
+KernelVersion:	6.4.0
+Contact:	[email protected]
+Description:	Show the process id of the application that opened the file. This is
+		helpful information for a monitor daemon that wants to kill the
+		application that opened the file.
drivers/dma/idxd/cdev.c +11 -0
index 9fc6565bf807..e124f0628f1c 100644
@@ -47,6 +47,7 @@ struct idxd_user_context {
 	struct idxd_dev idxd_dev;
 	u64 counters[COUNTER_MAX];
 	int id;
+	pid_t pid;
 };
 
 static void idxd_cdev_evl_drain_pasid(struct idxd_wq *wq, u32 pasid);
@@ -76,9 +77,18 @@ static ssize_t cr_fault_failures_show(struct device *dev,
 }
 static DEVICE_ATTR_RO(cr_fault_failures);
 
+static ssize_t pid_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	struct idxd_user_context *ctx = dev_to_uctx(dev);
+
+	return sysfs_emit(buf, "%u\n", ctx->pid);
+}
+static DEVICE_ATTR_RO(pid);
+
 static struct attribute *cdev_file_attributes[] = {
 	&dev_attr_cr_faults.attr,
 	&dev_attr_cr_fault_failures.attr,
+	&dev_attr_pid.attr,
 	NULL
 };
 
@@ -236,6 +246,7 @@ static int idxd_cdev_open(struct inode *inode, struct file *filp)
 
 	ctx->wq = wq;
 	filp->private_data = ctx;
+	ctx->pid = current->pid;
 
 	if (device_user_pasid_enabled(idxd)) {
 		sva = iommu_sva_bind_device(dev, current->mm);
Signature · 1 line
2.37.1
author
#24 lore raw

PATCH v2 16/16 dmaengine: idxd: add per wq PRS disable

In reply to Fenghua Yu

From: Dave Jiang <[email protected]>

Add sysfs knob for per wq Page Request Service disable. This knob disables PRS support for the specific wq. When this bit is set, it also overrides the wq's block on fault enabling.

---
 .../ABI/stable/sysfs-driver-dma-idxd          | 10 ++++
 drivers/dma/idxd/device.c                     |  6 +-
 drivers/dma/idxd/idxd.h                       |  1 +
 drivers/dma/idxd/registers.h                  |  5 +-
 drivers/dma/idxd/sysfs.c                      | 57 ++++++++++++++++++-
 5 files changed, 74 insertions(+), 5 deletions(-)
Documentation/ABI/stable/sysfs-driver-dma-idxd +10 -0
index 5d0df57f5298..534b7a3d59fc 100644
@@ -235,6 +235,16 @@ Contact:	[email protected]
 Description:	Indicate whether ATS disable is turned on for the workqueue.
 		0 indicates ATS is on, and 1 indicates ATS is off for the workqueue.
 
+What:		/sys/bus/dsa/devices/wq<m>.<n>/prs_disable
+Date:		Sept 14, 2022
+KernelVersion: 6.4.0
+Contact:	[email protected]
+Description:	Controls whether PRS disable is turned on for the workqueue.
+		0 indicates PRS is on, and 1 indicates PRS is off for the
+		workqueue. This option overrides block_on_fault attribute
+		if set. It's visible only on platforms that support the
+		capability.
+
 What:		/sys/bus/dsa/devices/wq<m>.<n>/occupancy
 Date		May 25, 2021
 KernelVersion:	5.14.0
drivers/dma/idxd/device.c +5 -1
index fd97b2b58734..3c80b9681c72 100644
@@ -967,12 +967,16 @@ static int idxd_wq_config_write(struct idxd_wq *wq)
 	wq->wqcfg->priority = wq->priority;
 
 	if (idxd->hw.gen_cap.block_on_fault &&
-	    test_bit(WQ_FLAG_BLOCK_ON_FAULT, &wq->flags))
+	    test_bit(WQ_FLAG_BLOCK_ON_FAULT, &wq->flags) &&
+	    !test_bit(WQ_FLAG_PRS_DISABLE, &wq->flags))
 		wq->wqcfg->bof = 1;
 
 	if (idxd->hw.wq_cap.wq_ats_support)
 		wq->wqcfg->wq_ats_disable = test_bit(WQ_FLAG_ATS_DISABLE, &wq->flags);
 
+	if (idxd->hw.wq_cap.wq_prs_support)
+		wq->wqcfg->wq_prs_disable = test_bit(WQ_FLAG_PRS_DISABLE, &wq->flags);
+
 	/* bytes 12-15 */
 	wq->wqcfg->max_xfer_shift = ilog2(wq->max_xfer_bytes);
 	idxd_wqcfg_set_max_batch_shift(idxd->data->type, wq->wqcfg, ilog2(wq->max_batch_size));
drivers/dma/idxd/idxd.h +1 -0
index b5d7ef611bae..3a20e4933d07 100644
@@ -143,6 +143,7 @@ enum idxd_wq_flag {
 	WQ_FLAG_DEDICATED = 0,
 	WQ_FLAG_BLOCK_ON_FAULT,
 	WQ_FLAG_ATS_DISABLE,
+	WQ_FLAG_PRS_DISABLE,
 };
 
 enum idxd_wq_type {
drivers/dma/idxd/registers.h +3 -2
index 9f3959d001b6..7b54a3939ea1 100644
@@ -59,7 +59,8 @@ union wq_cap_reg {
 		u64 occupancy:1;
 		u64 occupancy_int:1;
 		u64 op_config:1;
-		u64 rsvd3:9;
+		u64 wq_prs_support:1;
+		u64 rsvd4:8;
 	};
 	u64 bits;
 } __packed;
@@ -371,7 +372,7 @@ union wqcfg {
 		u32 mode:1;	/* shared or dedicated */
 		u32 bof:1;	/* block on fault */
 		u32 wq_ats_disable:1;
-		u32 rsvd2:1;
+		u32 wq_prs_disable:1;
 		u32 priority:4;
 		u32 pasid:20;
 		u32 pasid_en:1;
drivers/dma/idxd/sysfs.c +55 -2

89 lines not on this page · raw message

Signature · 1 line
2.37.1