Skip to contentSkip to conversation

s390/vfio-ap: dynamic configuration support

PATCH Superseded Tony Krowiak 3 participants 50 messages Feb 15, 2022 – Apr 6, 2022

Subsystems (5)

v18 of 20 Superseded Latest is v20 →

lore
Share & tools
lore

Tony Krowiak

author maintainer
#0 lore raw

v18 cover s390/vfio-ap: dynamic configuration support

The current design for AP pass-through does not support making dynamic changes to the AP matrix of a running guest resulting in a few deficiencies this patch series is intended to mitigate:

1. Adapters, domains and control domains can not be added to or removed from a running guest. In order to modify a guest's AP configuration, the guest must be terminated; only then can AP resources be assigned to or unassigned from the guest's matrix mdev. The new AP configuration becomes available to the guest when it is subsequently restarted.

2. The AP bus's /sys/bus/ap/apmask and /sys/bus/ap/aqmask interfaces can be modified by a root user without any restrictions. A change to either mask can result in AP queue devices being unbound from the vfio_ap device driver and bound to a zcrypt device driver even if a guest is using the queues, thus giving the host access to the guest's private crypto data and vice versa.

3. The APQNs derived from the Cartesian product of the APIDs of the adapters and APQIs of the domains assigned to a matrix mdev must reference an AP queue device bound to the vfio_ap device driver. The AP architecture allows assignment of AP resources that are not available to the system, so this artificial restriction is not compliant with the architecture.

4. The AP configuration profile can be dynamically changed for the linux host after a KVM guest is started. For example, a new domain can be dynamically added to the configuration profile via the SE or an HMC connected to a DPM enabled lpar. Likewise, AP adapters can be dynamically configured (online state) and deconfigured (standby state) using the SE, an SCLP command or an HMC connected to a DPM enabled lpar. This can result in inadvertent sharing of AP queues between the guest and host.

5. A root user can manually unbind an AP queue device representing a queue in use by a KVM guest via the vfio_ap device driver's sysfs unbind attribute. In this case, the guest will be using a queue that is not bound to the driver which violates the device model.

This patch series introduces the following changes to the current design to alleviate the shortcomings described above as well as to implement more of the AP architecture:

1. A root user will be prevented from making edits to the AP bus's /sys/bus/ap/apmask or /sys/bus/ap/aqmask if the change would transfer ownership of an APQN from the vfio_ap device driver to a zcrypt driver while the APQN is assigned to a matrix mdev.

2. Allow a root user to hot plug/unplug AP adapters, domains and control domains for a KVM guest using the matrix mdev via its sysfs assign/unassign attributes.

4. Allow assignment of an AP adapter or domain to a matrix mdev even if it results in assignment of an APQN that does not reference an AP queue device bound to the vfio_ap device driver, as long as the APQN is not reserved for use by the default zcrypt drivers (also known as over-provisioning of AP resources). Allowing over-provisioning of AP resources better models the architecture which does not preclude assigning AP resources that are not yet available in the system. Such APQNs, however, will not be assigned to the guest using the matrix mdev; only APQNs referencing AP queue devices bound to the vfio_ap device driver will actually get assigned to the guest.

5. Handle dynamic changes to the AP device model.

1. Rationale for changes to AP bus's apmask/aqmask interfaces:
----------------------------------------------------------
Due to the extremely sensitive nature of cryptographic data, it is imperative that great care be taken to ensure that such data is secured. Allowing a root user, either inadvertently or maliciously, to configure these masks such that a queue is shared between the host and a guest is not only avoidable, it is advisable. It was suggested that this scenario is better handled in user space with management software, but that does not preclude a malicious administrator from using the sysfs interfaces to gain access to a guest's crypto data. It was also suggested that this scenario could be avoided by taking access to the adapter away from the guest and zeroing out the queues prior to the vfio_ap driver releasing the device; however, stealing an adapter in use from a guest as a by-product of an operation is bad and will likely cause problems for the guest unnecessarily. It was decided that the most effective solution with the least number of negative side effects is to prevent the situation at the source.

2. Rationale for hot plug/unplug using matrix mdev sysfs interfaces:
----------------------------------------------------------------
Allowing a user to hot plug/unplug AP resources using the matrix mdev sysfs interfaces circumvents the need to terminate the guest in order to modify its AP configuration. Allowing dynamic configuration makes reconfiguring a guest's AP matrix much less disruptive.

3. Rationale for allowing over-provisioning of AP resources:
-----------------------------------------------------------
Allowing assignment of AP resources to a matrix mdev and ultimately to a guest better models the AP architecture. The architecture does not preclude assignment of unavailable AP resources. If a queue subsequently becomes available while a guest using the matrix mdev to which its APQN is assigned, the guest will be given access to it. If an APQN is dynamically unassigned from the underlying host system, it will automatically become unavailable to the guest.

Change log v17-v18:
------------------
* Added a new document, Documentation/s390/vfio-ap-locking.rst to describe the locking design for this series. This included a patch for adding the new doc to the MAITAINERS file for the VFIO AP maintainers.

* Added Reviewed-by for Halil in patch 6

* Restore filtering in the queue remove callback. (Halil)

* Added patch s390/vfio-ap: hot unplug of AP devices when mdev removed to unplug all AP devices from the guest when the mdev is removed.

* Split patch 9 (s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device) into xx patches:
- allow hot plug/unplug of AP devices when assigned/unassigned

* Replaced v17 patch 08/15, s390/vfio-ap: keep track of active guests with s390/vfio-ap: introduce new mutex to control access to the KVM pointer. No longer tracking guests with a list of struct ap_guest objects. Added a global mutex called kvm_lock which must be held whenever the kvm->lock is taken. (Halil)

* Changed signature of the filtering function to limit the APQNs examined. This change resulted in a cascade of changes to patches 5, 8 and 12. (Halil)

* Removed v17 patch 01/15 (s390/vfio-ap: Set pqap hook when vfio_ap module is loaded). (Halil, Jason).

* Split patch 14 (notify drivers on config changed and scan complete callbacks) into two patches: One to make the AP bus changes and the other to implement the callbacks in the vfio_ap device driver. This was done to facilitate merging the AP bus changes separately from the vfio_ap driver changes. (Harald)

* Removed check of driver flags in the __verify_card_reservations and __verify_queue_reservations functions in ap_bus.c to balance the weighting between the default and vfio_ap drivers. (Harald)

Tony Krowiak (17):
s390/ap: driver callback to indicate resource in use s390/ap: notify drivers on config changed and scan complete callbacks s390/vfio-ap: use new AP bus interface to search for queue devices s390/vfio-ap: move probe and remove callbacks to vfio_ap_ops.c s390/vfio-ap: manage link between queue struct and matrix mdev s390/vfio-ap: introduce shadow APCB
s390/vfio-ap: refresh guest's APCB by filtering APQNs assigned to mdev s390/vfio-ap: allow assignment of unavailable AP queues to mdev device s390/vfio-ap: introduce new mutex to control access to the KVM pointer s390/vfio-ap: allow hot plug/unplug of AP devices when
assigned/unassigned
s390/vfio-ap: hot plug/unplug of AP devices when probed/removed s390/vfio-ap: reset queues after adapter/domain unassignment s390/vfio-ap: implement in-use callback for vfio_ap driver s390/vfio-ap: sysfs attribute to display the guest's matrix s390/vfio-ap: handle config changed and scan complete notification s390/vfio-ap: update docs to include dynamic config support s390/Docs: new doc describing lock usage by the vfio_ap device driver
MAINTAINERS: pick up all vfio_ap docs for VFIO AP maintainers

 Documentation/s390/vfio-ap-locking.rst |  389 +++++++
 Documentation/s390/vfio-ap.rst         |  492 ++++++---
 MAINTAINERS                            |    2 +-
 drivers/s390/crypto/ap_bus.c           |  226 +++-
 drivers/s390/crypto/ap_bus.h           |   16 +
 drivers/s390/crypto/vfio_ap_drv.c      |   71 +-
 drivers/s390/crypto/vfio_ap_ops.c      | 1347 ++++++++++++++++++------
 drivers/s390/crypto/vfio_ap_private.h  |   57 +-
 8 files changed, 2058 insertions(+), 542 deletions(-)
 create mode 100644 Documentation/s390/vfio-ap-locking.rst
Signature · 1 line
2.31.1
author maintainer
#1 lore raw

PATCH v18 1/18 s390/ap: driver callback to indicate resource in use

Introduces a new driver callback to prevent a root user from re-assigning the APQN of a queue that is in use by a non-default host device driver to a default host device driver and vice versa. The callback will be invoked whenever a change to the AP bus's sysfs apmask or aqmask attributes would result in one or more APQNs being re-assigned. If the callback responds in the affirmative for any driver queried, the change to the apmask or aqmask will be rejected with a device busy error.

For this patch, only non-default drivers will be queried. Currently, there is only one non-default driver, the vfio_ap device driver. The vfio_ap device driver facilitates pass-through of an AP queue to a guest. The idea here is that a guest may be administered by a different sysadmin than the host and we don't want AP resources to unexpectedly disappear from a guest's AP configuration (i.e., adapters and domains assigned to the matrix mdev). This will enforce the proper procedure for removing AP resources intended for guest usage which is to first unassign them from the matrix mdev, then unbind them from the vfio_ap device driver.

---
 drivers/s390/crypto/ap_bus.c | 145 ++++++++++++++++++++++++++++++++---
 drivers/s390/crypto/ap_bus.h |   4 +
 2 files changed, 139 insertions(+), 10 deletions(-)
drivers/s390/crypto/ap_bus.c +135 -10
index 1986243f9cd3..d71d2d2c341f 100644
@@ -36,6 +36,7 @@
 #include <linux/mod_devicetable.h>
 #include <linux/debugfs.h>
 #include <linux/ctype.h>
+#include <linux/module.h>
 
 #include "ap_bus.h"
 #include "ap_debug.h"
@@ -1067,6 +1068,23 @@ static int modify_bitmap(const char *str, unsigned long *bitmap, int bits)
 	return 0;
 }
 
+static int ap_parse_bitmap_str(const char *str, unsigned long *bitmap, int bits,
+			       unsigned long *newmap)
+{
+	unsigned long size;
+	int rc;
+
+	size = BITS_TO_LONGS(bits) * sizeof(unsigned long);
+	if (*str == '+' || *str == '-') {
+		memcpy(newmap, bitmap, size);
+		rc = modify_bitmap(str, newmap, bits);
+	} else {
+		memset(newmap, 0, size);
+		rc = hex2bitmap(str, newmap, bits);
+	}
+	return rc;
+}
+
 int ap_parse_mask_str(const char *str,
 		      unsigned long *bitmap, int bits,
 		      struct mutex *lock)
@@ -1086,14 +1104,7 @@ int ap_parse_mask_str(const char *str,
 		kfree(newmap);
 		return -ERESTARTSYS;
 	}
-
-	if (*str == '+' || *str == '-') {
-		memcpy(newmap, bitmap, size);
-		rc = modify_bitmap(str, newmap, bits);
-	} else {
-		memset(newmap, 0, size);
-		rc = hex2bitmap(str, newmap, bits);
-	}
+	rc = ap_parse_bitmap_str(str, bitmap, bits, newmap);
 	if (rc == 0)
 		memcpy(bitmap, newmap, size);
 	mutex_unlock(lock);
@@ -1286,12 +1297,69 @@ static ssize_t apmask_show(struct bus_type *bus, char *buf)
 	return rc;
 }
 
+static int __verify_card_reservations(struct device_driver *drv, void *data)
+{
+	int rc = 0;
+	struct ap_driver *ap_drv = to_ap_drv(drv);
+	unsigned long *newapm = (unsigned long *)data;
+
+	/*
+	 * increase the driver's module refcounter to be sure it is not
+	 * going away when we invoke the callback function.
+	 */
+	if (!try_module_get(drv->owner))
+		return 0;
+
+	if (ap_drv->in_use) {
+		rc = ap_drv->in_use(newapm, ap_perms.aqm);
+		if (rc)
+			rc = -EBUSY;
+	}
+
+	/* release the driver's module */
+	module_put(drv->owner);
+
+	return rc;
+}
+
+static int apmask_commit(unsigned long *newapm)
+{
+	int rc;
+	unsigned long reserved[BITS_TO_LONGS(AP_DEVICES)];
+
+	/*
+	 * Check if any bits in the apmask have been set which will
+	 * result in queues being removed from non-default drivers
+	 */
+	if (bitmap_andnot(reserved, newapm, ap_perms.apm, AP_DEVICES)) {
+		rc = bus_for_each_drv(&ap_bus_type, NULL, reserved,
+				      __verify_card_reservations);
+		if (rc)
+			return rc;
+	}
+
+	memcpy(ap_perms.apm, newapm, APMASKSIZE);
+
+	return 0;
+}
+
 static ssize_t apmask_store(struct bus_type *bus, const char *buf,
 			    size_t count)
 {
 	int rc;
+	DECLARE_BITMAP(newapm, AP_DEVICES);
+
+	if (mutex_lock_interruptible(&ap_perms_mutex))
+		return -ERESTARTSYS;
 
-	rc = ap_parse_mask_str(buf, ap_perms.apm, AP_DEVICES, &ap_perms_mutex);
+	rc = ap_parse_bitmap_str(buf, ap_perms.apm, AP_DEVICES, newapm);
+	if (rc)
+		goto done;
+
+	rc = apmask_commit(newapm);
+
+done:
+	mutex_unlock(&ap_perms_mutex);
 	if (rc)
 		return rc;
 
@@ -1317,12 +1385,69 @@ static ssize_t aqmask_show(struct bus_type *bus, char *buf)
 	return rc;
 }
 
+static int __verify_queue_reservations(struct device_driver *drv, void *data)
+{
+	int rc = 0;
+	struct ap_driver *ap_drv = to_ap_drv(drv);
+	unsigned long *newaqm = (unsigned long *)data;
+
+	/*
+	 * increase the driver's module refcounter to be sure it is not
+	 * going away when we invoke the callback function.
+	 */
+	if (!try_module_get(drv->owner))
+		return 0;
+
+	if (ap_drv->in_use) {
+		rc = ap_drv->in_use(ap_perms.apm, newaqm);
+		if (rc)
+			return -EBUSY;
+	}
+
+	/* release the driver's module */
+	module_put(drv->owner);
+
+	return rc;
+}
+
+static int aqmask_commit(unsigned long *newaqm)
+{
+	int rc;
+	unsigned long reserved[BITS_TO_LONGS(AP_DOMAINS)];
+
+	/*
+	 * Check if any bits in the aqmask have been set which will
+	 * result in queues being removed from non-default drivers
+	 */
+	if (bitmap_andnot(reserved, newaqm, ap_perms.aqm, AP_DOMAINS)) {
+		rc = bus_for_each_drv(&ap_bus_type, NULL, reserved,
+				      __verify_queue_reservations);
+		if (rc)
+			return rc;
+	}
+
+	memcpy(ap_perms.aqm, newaqm, AQMASKSIZE);
+
+	return 0;
+}
+
 static ssize_t aqmask_store(struct bus_type *bus, const char *buf,
 			    size_t count)
 {
 	int rc;
+	DECLARE_BITMAP(newaqm, AP_DOMAINS);
 
-	rc = ap_parse_mask_str(buf, ap_perms.aqm, AP_DOMAINS, &ap_perms_mutex);
+	if (mutex_lock_interruptible(&ap_perms_mutex))
+		return -ERESTARTSYS;
+
+	rc = ap_parse_bitmap_str(buf, ap_perms.aqm, AP_DOMAINS, newaqm);
+	if (rc)
+		goto done;
+
+	rc = aqmask_commit(newaqm);
+
+done:
+	mutex_unlock(&ap_perms_mutex);
 	if (rc)
 		return rc;
 
drivers/s390/crypto/ap_bus.h +4 -0
index 95b577754b35..67c1bef60ad5 100644
@@ -142,6 +142,7 @@ struct ap_driver {
 
 	int (*probe)(struct ap_device *);
 	void (*remove)(struct ap_device *);
+	int (*in_use)(unsigned long *apm, unsigned long *aqm);
 };
 
 #define to_ap_drv(x) container_of((x), struct ap_driver, driver)
@@ -289,6 +290,9 @@ void ap_queue_init_state(struct ap_queue *aq);
 struct ap_card *ap_card_create(int id, int queue_depth, int raw_type,
 			       int comp_type, unsigned int functions, int ml);
 
+#define APMASKSIZE (BITS_TO_LONGS(AP_DEVICES) * sizeof(unsigned long))
+#define AQMASKSIZE (BITS_TO_LONGS(AP_DOMAINS) * sizeof(unsigned long))
+
 struct ap_perms {
 	unsigned long ioctlm[BITS_TO_LONGS(AP_IOCTLS)];
 	unsigned long apm[BITS_TO_LONGS(AP_DEVICES)];
Signature · 1 line
2.31.1
author maintainer
#6 lore raw

PATCH v18 6/18 s390/vfio-ap: introduce shadow APCB

In reply to Tony Krowiak

The APCB is a field within the CRYCB that provides the AP configuration to a KVM guest. Let's introduce a shadow copy of the KVM guest's APCB and maintain it for the lifespan of the guest.

The shadow APCB serves the following purposes:

1. The shadow APCB can be maintained even when the mediated device is not currently in use by a KVM guest. Since the mediated device's AP configuration is filtered to ensure that no AP queues are passed through to the KVM guest that are not bound to the vfio_ap device driver or available to the host, the mediated device's AP configuration may differ from the guest's. Having a shadow of a guest's APCB allows us to provide a sysfs interface to view the guest's APCB even if the mediated device is not currently passed through to a KVM guest. This can aid in problem determination when the guest is unexpectedly missing AP resources.

2. If filtering was done in-place for the real APCB, the guest could pick up a transient state. Doing the filtering on a shadow and transferring the AP configuration to the real APCB after the guest is started or when AP resources are assigned to or unassigned from the mediated device, or when the host configuration changes, the guest's AP configuration will never be in a transient state.

---
 drivers/s390/crypto/vfio_ap_ops.c     | 10 ++++++----
 drivers/s390/crypto/vfio_ap_private.h |  2 ++
 2 files changed, 8 insertions(+), 4 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +6 -4
index aa838ed47406..4b676a55f203 100644
@@ -336,6 +336,7 @@ static int vfio_ap_mdev_probe(struct mdev_device *mdev)
 	matrix_mdev->mdev = mdev;
 	vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix);
 	matrix_mdev->pqap_hook = handle_pqap;
+	vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_apcb);
 	hash_init(matrix_mdev->qtable.queues);
 	mdev_set_drvdata(mdev, matrix_mdev);
 	mutex_lock(&matrix_dev->lock);
@@ -1185,10 +1186,11 @@ static int vfio_ap_mdev_set_kvm(struct ap_matrix_mdev *matrix_mdev,
 
 		kvm_get_kvm(kvm);
 		matrix_mdev->kvm = kvm;
-		kvm_arch_crypto_set_masks(kvm,
-					  matrix_mdev->matrix.apm,
-					  matrix_mdev->matrix.aqm,
-					  matrix_mdev->matrix.adm);
+		memcpy(&matrix_mdev->shadow_apcb, &matrix_mdev->matrix,
+		       sizeof(struct ap_matrix));
+		kvm_arch_crypto_set_masks(kvm, matrix_mdev->shadow_apcb.apm,
+					  matrix_mdev->shadow_apcb.aqm,
+					  matrix_mdev->shadow_apcb.adm);
 
 		mutex_unlock(&kvm->lock);
 		mutex_unlock(&matrix_dev->lock);
drivers/s390/crypto/vfio_ap_private.h +2 -0
index aea6a8b854b3..fa11a7e91e24 100644
@@ -91,6 +91,7 @@ struct ap_queue_table {
  * @node:	allows the ap_matrix_mdev struct to be added to a list
  * @matrix:	the adapters, usage domains and control domains assigned to the
  *		mediated matrix device.
+ * @shadow_apcb:    the shadow copy of the APCB field of the KVM guest's CRYCB
  * @group_notifier: notifier block used for specifying callback function for
  *		    handling the VFIO_GROUP_NOTIFY_SET_KVM event
  * @iommu_notifier: notifier block used for specifying callback function for
@@ -105,6 +106,7 @@ struct ap_matrix_mdev {
 	struct vfio_device vdev;
 	struct list_head node;
 	struct ap_matrix matrix;
+	struct ap_matrix shadow_apcb;
 	struct notifier_block group_notifier;
 	struct notifier_block iommu_notifier;
 	struct kvm *kvm;
Signature · 1 line
2.31.1
author maintainer
#3 lore raw

PATCH v18 3/18 s390/vfio-ap: use new AP bus interface to search for queue devices

In reply to Tony Krowiak

This patch refactors the vfio_ap device driver to use the AP bus's ap_get_qdev() function to retrieve the vfio_ap_queue struct containing information about a queue that is bound to the vfio_ap device driver. The bus's ap_get_qdev() function retrieves the queue device from a hashtable keyed by APQN. This is much more efficient than looping over the list of devices attached to the AP bus by several orders of magnitude.

---
 drivers/s390/crypto/vfio_ap_ops.c | 23 +++++++++--------------
 1 file changed, 9 insertions(+), 14 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +9 -14
index abc0b9b88386..2b6b61719cce 100644
@@ -28,13 +28,6 @@ static int vfio_ap_mdev_reset_queues(struct ap_matrix_mdev *matrix_mdev);
 static struct vfio_ap_queue *vfio_ap_find_queue(int apqn);
 static const struct vfio_device_ops vfio_ap_matrix_dev_ops;
 
-static int match_apqn(struct device *dev, const void *data)
-{
-	struct vfio_ap_queue *q = dev_get_drvdata(dev);
-
-	return (q->apqn == *(int *)(data)) ? 1 : 0;
-}
-
 /**
  * vfio_ap_get_queue - retrieve a queue with a specific APQN from a list
  * @matrix_mdev: the associated mediated matrix
@@ -1193,15 +1186,17 @@ static int vfio_ap_mdev_group_notifier(struct notifier_block *nb,
 
 static struct vfio_ap_queue *vfio_ap_find_queue(int apqn)
 {
-	struct device *dev;
+	struct ap_queue *queue;
 	struct vfio_ap_queue *q = NULL;
 
-	dev = driver_find_device(&matrix_dev->vfio_ap_drv->driver, NULL,
-				 &apqn, match_apqn);
-	if (dev) {
-		q = dev_get_drvdata(dev);
-		put_device(dev);
-	}
+	queue = ap_get_qdev(apqn);
+	if (!queue)
+		return NULL;
+
+	if (queue->ap_dev.device.driver == &matrix_dev->vfio_ap_drv->driver)
+		q = dev_get_drvdata(&queue->ap_dev.device);
+
+	put_device(&queue->ap_dev.device);
 
 	return q;
 }
Signature · 1 line
2.31.1
author maintainer
#2 lore raw

PATCH v18 2/18 s390/ap: notify drivers on config changed and scan complete callbacks

In reply to Tony Krowiak

This patch introduces an extension to the ap bus to notify device drivers when the host AP configuration changes - i.e., adapters, domains or control domains are added or removed. When an adapter or domain is added to the host's AP configuration, the AP bus will create the associated queue devices in the linux sysfs device model. Each new type 10 (i.e., CEX4) or newer queue device with an APQN that is not reserved for the default device driver will get bound to the vfio_ap device driver. Likewise, whan an adapter or domain is removed from the host's AP configuration, the AP bus will remove the associated queue devices from the sysfs device model. Each of the queues that is bound to the vfio_ap device driver will get unbound.

With the introduction of hot plug support, binding or unbinding of a queue device will result in plugging or unplugging one or more queues from a guest that is using the queue. If there are multiple changes to the host's AP configuration, it could result in the probe and remove callbacks getting invoked multiple times. Each time queues are plugged into or unplugged from a guest, the guest's VCPUs must be taken out of SIE. If this occurs multiple times due to changes in the host's AP configuration, that can have an undesirable negative affect on the guest's performance.

To alleviate this problem, this patch introduces two new callbacks: one to notify the vfio_ap device driver when the AP bus scan routine detects a change to the host's AP configuration; and, one to notify the driver when the AP bus is done scanning. This will allow the vfio_ap driver to do bulk processing of all affected adapters, domains and control domains for affected guests rather than plugging or unplugging them one at a time when the probe or remove callback is invoked. The two new callbacks are:

void (*on_config_changed)(struct ap_config_info *new_config_info,

                          struct ap_config_info *old_config_info);

This callback is invoked at the start of the AP bus scan function when it determines that the host AP configuration information has changed since the previous scan. This is done by storing an old and current QCI info struct and comparing them. If there is any difference, the callback is invoked.

void (*on_scan_complete)(struct ap_config_info *new_config_info,

                         struct ap_config_info *old_config_info);

The on_scan_complete callback is invoked after the ap bus scan is completed if the host AP configuration data has changed.

---
 drivers/s390/crypto/ap_bus.c | 81 +++++++++++++++++++++++++++++++++++-
 drivers/s390/crypto/ap_bus.h | 12 ++++++
 2 files changed, 91 insertions(+), 2 deletions(-)
drivers/s390/crypto/ap_bus.c +79 -2
index d71d2d2c341f..f5fae8b62bdf 100644
@@ -92,6 +92,7 @@ static atomic64_t ap_bindings_complete_count = ATOMIC64_INIT(0);
 static DECLARE_COMPLETION(ap_init_apqn_bindings_complete);
 
 static struct ap_config_info *ap_qci_info;
+static struct ap_config_info *ap_qci_info_old;
 
 /*
  * AP bus related debug feature things.
@@ -229,9 +230,14 @@ static void __init ap_init_qci_info(void)
 	ap_qci_info = kzalloc(sizeof(*ap_qci_info), GFP_KERNEL);
 	if (!ap_qci_info)
 		return;
+	ap_qci_info_old = kzalloc(sizeof(*ap_qci_info_old), GFP_KERNEL);
+	if (!ap_qci_info_old)
+		return;
 	if (ap_fetch_qci_info(ap_qci_info) != 0) {
 		kfree(ap_qci_info);
+		kfree(ap_qci_info_old);
 		ap_qci_info = NULL;
+		ap_qci_info_old = NULL;
 		return;
 	}
 	AP_DBF_INFO("%s successful fetched initial qci info\n", __func__);
@@ -248,6 +254,8 @@ static void __init ap_init_qci_info(void)
 				    __func__, ap_max_domain_id);
 		}
 	}
+
+	memcpy(ap_qci_info_old, ap_qci_info, sizeof(*ap_qci_info));
 }
 
 /*
@@ -1630,6 +1638,49 @@ static int __match_queue_device_with_queue_id(struct device *dev, const void *da
 		&& AP_QID_QUEUE(to_ap_queue(dev)->qid) == (int)(long) data;
 }
 
+/* Helper function for notify_config_changed */
+static int __drv_notify_config_changed(struct device_driver *drv, void *data)
+{
+	struct ap_driver *ap_drv = to_ap_drv(drv);
+
+	if (try_module_get(drv->owner)) {
+		if (ap_drv->on_config_changed)
+			ap_drv->on_config_changed(ap_qci_info, ap_qci_info_old);
+		module_put(drv->owner);
+	}
+
+	return 0;
+}
+
+/* Notify all drivers about an qci config change */
+static inline void notify_config_changed(void)
+{
+	bus_for_each_drv(&ap_bus_type, NULL, NULL,
+			 __drv_notify_config_changed);
+}
+
+/* Helper function for notify_scan_complete */
+static int __drv_notify_scan_complete(struct device_driver *drv, void *data)
+{
+	struct ap_driver *ap_drv = to_ap_drv(drv);
+
+	if (try_module_get(drv->owner)) {
+		if (ap_drv->on_scan_complete)
+			ap_drv->on_scan_complete(ap_qci_info,
+						 ap_qci_info_old);
+		module_put(drv->owner);
+	}
+
+	return 0;
+}
+
+/* Notify all drivers about bus scan complete */
+static inline void notify_scan_complete(void)
+{
+	bus_for_each_drv(&ap_bus_type, NULL, NULL,
+			 __drv_notify_scan_complete);
+}
+
 /*
  * Helper function for ap_scan_bus().
  * Remove card device and associated queue devices.
@@ -1917,6 +1968,25 @@ static inline void ap_scan_adapter(int ap)
 	put_device(&ac->ap_dev.device);
 }
 
+/**
+ * ap_get_configuration - get the host AP configuration
+ *
+ * Stores the host AP configuration information returned from the previous call
+ * to Query Configuration Information (QCI), then retrieves and stores the
+ * current AP configuration returned from QCI.
+ *
+ * Return: true if the host AP configuration changed between calls to QCI;
+ * otherwise, return false.
+ */
+static bool ap_get_configuration(void)
+{
+	memcpy(ap_qci_info_old, ap_qci_info, sizeof(*ap_qci_info));
+	ap_fetch_qci_info(ap_qci_info);
+
+	return memcmp(ap_qci_info, ap_qci_info_old,
+		      sizeof(struct ap_config_info)) != 0;
+}
+
 /**
  * ap_scan_bus(): Scan the AP bus for new devices
  * Runs periodically, workqueue timer (ap_config_time)
@@ -1924,9 +1994,12 @@ static inline void ap_scan_adapter(int ap)
  */
 static void ap_scan_bus(struct work_struct *unused)
 {
-	int ap;
+	int ap, config_changed = 0;
 
-	ap_fetch_qci_info(ap_qci_info);
+	/* config change notify */
+	config_changed = ap_get_configuration();
+	if (config_changed)
+		notify_config_changed();
 	ap_select_domain();
 
 	AP_DBF_DBG("%s running\n", __func__);
@@ -1935,6 +2008,10 @@ static void ap_scan_bus(struct work_struct *unused)
 	for (ap = 0; ap <= ap_max_adapter_id; ap++)
 		ap_scan_adapter(ap);
 
+	/* scan complete notify */
+	if (config_changed)
+		notify_scan_complete();
+
 	/* check if there is at least one queue available with default domain */
 	if (ap_domain_index >= 0) {
 		struct device *dev =
drivers/s390/crypto/ap_bus.h +12 -0
index 67c1bef60ad5..4de062ea6b76 100644
@@ -143,6 +143,18 @@ struct ap_driver {
 	int (*probe)(struct ap_device *);
 	void (*remove)(struct ap_device *);
 	int (*in_use)(unsigned long *apm, unsigned long *aqm);
+	/*
+	 * Called at the start of the ap bus scan function when
+	 * the crypto config information (qci) has changed.
+	 */
+	void (*on_config_changed)(struct ap_config_info *new_config_info,
+				  struct ap_config_info *old_config_info);
+	/*
+	 * Called at the end of the ap bus scan function when
+	 * the crypto config information (qci) has changed.
+	 */
+	void (*on_scan_complete)(struct ap_config_info *new_config_info,
+				 struct ap_config_info *old_config_info);
 };
 
 #define to_ap_drv(x) container_of((x), struct ap_driver, driver)
Signature · 1 line
2.31.1
author maintainer
#4 lore raw

PATCH v18 4/18 s390/vfio-ap: move probe and remove callbacks to vfio_ap_ops.c

In reply to Tony Krowiak

Let's move the probe and remove callbacks into the vfio_ap_ops.c file to keep all code related to managing queues in a single file. This way, all functions related to queue management can be removed from the vfio_ap_private.h header file defining the public interfaces for the vfio_ap device driver.

---
 drivers/s390/crypto/vfio_ap_drv.c     | 59 +--------------------------
 drivers/s390/crypto/vfio_ap_ops.c     | 31 +++++++++++++-
 drivers/s390/crypto/vfio_ap_private.h |  5 ++-
 3 files changed, 34 insertions(+), 61 deletions(-)
drivers/s390/crypto/vfio_ap_drv.c +2 -57
index e043ae236630..5013a5531171 100644
@@ -100,64 +100,9 @@ static const struct attribute_group vfio_queue_attr_group = {
 	.attrs = vfio_queue_attrs,
 };
 
-/**
- * vfio_ap_queue_dev_probe: Allocate a vfio_ap_queue structure and associate it
- *			    with the device as driver_data.
- *
- * @apdev: the AP device being probed
- *
- * Return: returns 0 if the probe succeeded; otherwise, returns an error if
- *	   storage could not be allocated for a vfio_ap_queue object or the
- *	   sysfs 'status' attribute could not be created for the queue device.
- */
-static int vfio_ap_queue_dev_probe(struct ap_device *apdev)
-{
-	int ret;
-	struct vfio_ap_queue *q;
-
-	q = kzalloc(sizeof(*q), GFP_KERNEL);
-	if (!q)
-		return -ENOMEM;
-
-	mutex_lock(&matrix_dev->lock);
-	dev_set_drvdata(&apdev->device, q);
-	q->apqn = to_ap_queue(&apdev->device)->qid;
-	q->saved_isc = VFIO_AP_ISC_INVALID;
-
-	ret = sysfs_create_group(&apdev->device.kobj, &vfio_queue_attr_group);
-	if (ret) {
-		dev_set_drvdata(&apdev->device, NULL);
-		kfree(q);
-	}
-
-	mutex_unlock(&matrix_dev->lock);
-
-	return ret;
-}
-
-/**
- * vfio_ap_queue_dev_remove: Free the associated vfio_ap_queue structure.
- *
- * @apdev: the AP device being removed
- *
- * Takes the matrix lock to avoid actions on this device while doing the remove.
- */
-static void vfio_ap_queue_dev_remove(struct ap_device *apdev)
-{
-	struct vfio_ap_queue *q;
-
-	mutex_lock(&matrix_dev->lock);
-	sysfs_remove_group(&apdev->device.kobj, &vfio_queue_attr_group);
-	q = dev_get_drvdata(&apdev->device);
-	vfio_ap_mdev_reset_queue(q, 1);
-	dev_set_drvdata(&apdev->device, NULL);
-	kfree(q);
-	mutex_unlock(&matrix_dev->lock);
-}
-
 static struct ap_driver vfio_ap_drv = {
-	.probe = vfio_ap_queue_dev_probe,
-	.remove = vfio_ap_queue_dev_remove,
+	.probe = vfio_ap_mdev_probe_queue,
+	.remove = vfio_ap_mdev_remove_queue,
 	.ids = ap_queue_ids,
 };
 
drivers/s390/crypto/vfio_ap_ops.c +29 -2
index 2b6b61719cce..c963443873e1 100644
@@ -1201,8 +1201,7 @@ static struct vfio_ap_queue *vfio_ap_find_queue(int apqn)
 	return q;
 }
 
-int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q,
-			     unsigned int retry)
+static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, unsigned int retry)
 {
 	struct ap_queue_status status;
 	int ret;
@@ -1411,3 +1410,31 @@ void vfio_ap_mdev_unregister(void)
 	mdev_unregister_device(&matrix_dev->device);
 	mdev_unregister_driver(&vfio_ap_matrix_driver);
 }
+
+int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
+{
+	struct vfio_ap_queue *q;
+
+	q = kzalloc(sizeof(*q), GFP_KERNEL);
+	if (!q)
+		return -ENOMEM;
+	mutex_lock(&matrix_dev->lock);
+	q->apqn = to_ap_queue(&apdev->device)->qid;
+	q->saved_isc = VFIO_AP_ISC_INVALID;
+	dev_set_drvdata(&apdev->device, q);
+	mutex_unlock(&matrix_dev->lock);
+
+	return 0;
+}
+
+void vfio_ap_mdev_remove_queue(struct ap_device *apdev)
+{
+	struct vfio_ap_queue *q;
+
+	mutex_lock(&matrix_dev->lock);
+	q = dev_get_drvdata(&apdev->device);
+	vfio_ap_mdev_reset_queue(q, 1);
+	dev_set_drvdata(&apdev->device, NULL);
+	kfree(q);
+	mutex_unlock(&matrix_dev->lock);
+}
drivers/s390/crypto/vfio_ap_private.h +3 -2
index 648fcaf8104a..3cade25a1620 100644
@@ -119,7 +119,8 @@ struct vfio_ap_queue {
 
 int vfio_ap_mdev_register(void);
 void vfio_ap_mdev_unregister(void);
-int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q,
-			     unsigned int retry);
+
+int vfio_ap_mdev_probe_queue(struct ap_device *queue);
+void vfio_ap_mdev_remove_queue(struct ap_device *queue);
 
 #endif /* _VFIO_AP_PRIVATE_H_ */
Signature · 1 line
2.31.1
author maintainer
#5 lore raw

PATCH v18 5/18 s390/vfio-ap: manage link between queue struct and matrix mdev

In reply to Tony Krowiak

Let's create links between each queue device bound to the vfio_ap device driver and the matrix mdev to which the queue's APQN is assigned. The idea is to facilitate efficient retrieval of the objects representing the queue devices and matrix mdevs as well as to verify that a queue assigned to a matrix mdev is bound to the driver.

The links will be created as follows:

* When the queue device is probed, if its APQN is assigned to a matrix mdev, the structures representing the queue device and the matrix mdev will be linked.

* When an adapter or domain is assigned to a matrix mdev, for each new APQN assigned that references a queue device bound to the vfio_ap device driver, the structures representing the queue device and the matrix mdev will be linked.

The links will be removed as follows:

* When the queue device is removed, if its APQN is assigned to a matrix mdev, the link from the structure representing the matrix mdev to the structure representing the queue will be removed. The link from the queue to the matrix mdev will be maintained because if the queue device is being removed due to a manual sysfs unbind, it may be needed after the queue is reset to clean up the IRQ resources allocated to enable AP interrupts for the KVM guest. Since the storage for the structure representing the queue device is ultimately freed by the remove callback, keeping the reference shouldn't be a problem.

* When an adapter or domain is unassigned from a matrix mdev, for each APQN unassigned that references a queue device bound to the vfio_ap device driver, the structures representing the queue device and the matrix mdev will be unlinked.

* When an mdev is removed, the link from any queues assigned to the mdev to the mdev will be removed.

---
 drivers/s390/crypto/vfio_ap_ops.c     | 192 +++++++++++++++++++++-----
 drivers/s390/crypto/vfio_ap_private.h |  14 ++
 2 files changed, 169 insertions(+), 37 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +155 -37
index c963443873e1..aa838ed47406 100644
@@ -29,32 +29,27 @@ static struct vfio_ap_queue *vfio_ap_find_queue(int apqn);
 static const struct vfio_device_ops vfio_ap_matrix_dev_ops;
 
 /**
- * vfio_ap_get_queue - retrieve a queue with a specific APQN from a list
- * @matrix_mdev: the associated mediated matrix
- * @apqn: The queue APQN
+ * vfio_ap_mdev_get_queue - retrieve a queue with a specific APQN from a
+ *			    hash table of queues assigned to a matrix mdev
+ * @matrix_mdev: the matrix mdev
+ * @apqn: The APQN of a queue device
  *
- * Retrieve a queue with a specific APQN from the list of the
- * devices of the vfio_ap_drv.
- * Verify that the APID and the APQI are set in the matrix.
- *
- * Return: the pointer to the associated vfio_ap_queue
+ * Return: the pointer to the vfio_ap_queue struct representing the queue or
+ *	   NULL if the queue is not assigned to @matrix_mdev
  */
-static struct vfio_ap_queue *vfio_ap_get_queue(
+static struct vfio_ap_queue *vfio_ap_mdev_get_queue(
 					struct ap_matrix_mdev *matrix_mdev,
 					int apqn)
 {
 	struct vfio_ap_queue *q;
 
-	if (!test_bit_inv(AP_QID_CARD(apqn), matrix_mdev->matrix.apm))
-		return NULL;
-	if (!test_bit_inv(AP_QID_QUEUE(apqn), matrix_mdev->matrix.aqm))
-		return NULL;
-
-	q = vfio_ap_find_queue(apqn);
-	if (q)
-		q->matrix_mdev = matrix_mdev;
+	hash_for_each_possible(matrix_mdev->qtable.queues, q, mdev_qnode,
+			       apqn) {
+		if (q && q->apqn == apqn)
+			return q;
+	}
 
-	return q;
+	return NULL;
 }
 
 /**
@@ -172,7 +167,6 @@ static struct ap_queue_status vfio_ap_irq_disable(struct vfio_ap_queue *q)
 		  status.response_code);
 end_free:
 	vfio_ap_free_aqic_resources(q);
-	q->matrix_mdev = NULL;
 	return status;
 }
 
@@ -295,7 +289,7 @@ static int handle_pqap(struct kvm_vcpu *vcpu)
 	if (!matrix_mdev->kvm)
 		goto out_unlock;
 
-	q = vfio_ap_get_queue(matrix_mdev, apqn);
+	q = vfio_ap_mdev_get_queue(matrix_mdev, apqn);
 	if (!q)
 		goto out_unlock;
 
@@ -342,6 +336,8 @@ static int vfio_ap_mdev_probe(struct mdev_device *mdev)
 	matrix_mdev->mdev = mdev;
 	vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix);
 	matrix_mdev->pqap_hook = handle_pqap;
+	hash_init(matrix_mdev->qtable.queues);
+	mdev_set_drvdata(mdev, matrix_mdev);
 	mutex_lock(&matrix_dev->lock);
 	list_add(&matrix_mdev->node, &matrix_dev->mdev_list);
 	mutex_unlock(&matrix_dev->lock);
@@ -363,6 +359,55 @@ static int vfio_ap_mdev_probe(struct mdev_device *mdev)
 	return ret;
 }
 
+static void vfio_ap_mdev_link_queue(struct ap_matrix_mdev *matrix_mdev,
+				    struct vfio_ap_queue *q)
+{
+	if (q) {
+		q->matrix_mdev = matrix_mdev;
+		hash_add(matrix_mdev->qtable.queues, &q->mdev_qnode, q->apqn);
+	}
+}
+
+static void vfio_ap_mdev_link_apqn(struct ap_matrix_mdev *matrix_mdev, int apqn)
+{
+	struct vfio_ap_queue *q;
+
+	q = vfio_ap_find_queue(apqn);
+	vfio_ap_mdev_link_queue(matrix_mdev, q);
+}
+
+static void vfio_ap_unlink_queue_fr_mdev(struct vfio_ap_queue *q)
+{
+	hash_del(&q->mdev_qnode);
+}
+
+static void vfio_ap_unlink_mdev_fr_queue(struct vfio_ap_queue *q)
+{
+	q->matrix_mdev = NULL;
+}
+
+static void vfio_ap_mdev_unlink_queue(struct vfio_ap_queue *q)
+{
+	vfio_ap_unlink_queue_fr_mdev(q);
+	vfio_ap_unlink_mdev_fr_queue(q);
+}
+
+static void vfio_ap_mdev_unlink_fr_queues(struct ap_matrix_mdev *matrix_mdev)
+{
+	struct vfio_ap_queue *q;
+	unsigned long apid, apqi;
+
+	for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) {
+		for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm,
+				     AP_DOMAINS) {
+			q = vfio_ap_mdev_get_queue(matrix_mdev,
+						   AP_MKQID(apid, apqi));
+			if (q)
+				q->matrix_mdev = NULL;
+		}
+	}
+}
+
 static void vfio_ap_mdev_remove(struct mdev_device *mdev)
 {
 	struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(&mdev->dev);
@@ -371,6 +416,7 @@ static void vfio_ap_mdev_remove(struct mdev_device *mdev)
 
 	mutex_lock(&matrix_dev->lock);
 	vfio_ap_mdev_reset_queues(matrix_mdev);
+	vfio_ap_mdev_unlink_fr_queues(matrix_mdev);
 	list_del(&matrix_mdev->node);
 	mutex_unlock(&matrix_dev->lock);
 	vfio_uninit_group_dev(&matrix_mdev->vdev);
@@ -579,6 +625,16 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
 	return 0;
 }
 
+static void vfio_ap_mdev_link_adapter(struct ap_matrix_mdev *matrix_mdev,
+				      unsigned long apid)
+{
+	unsigned long apqi;
+
+	for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS)
+		vfio_ap_mdev_link_apqn(matrix_mdev,
+				       AP_MKQID(apid, apqi));
+}
+
 /**
  * assign_adapter_store - parses the APID from @buf and sets the
  * corresponding bit in the mediated matrix device's APM
@@ -649,6 +705,7 @@ static ssize_t assign_adapter_store(struct device *dev,
 	if (ret)
 		goto share_err;
 
+	vfio_ap_mdev_link_adapter(matrix_mdev, apid);
 	ret = count;
 	goto done;
 
@@ -661,6 +718,20 @@ static ssize_t assign_adapter_store(struct device *dev,
 }
 static DEVICE_ATTR_WO(assign_adapter);
 
+static void vfio_ap_mdev_unlink_adapter(struct ap_matrix_mdev *matrix_mdev,
+					unsigned long apid)
+{
+	unsigned long apqi;
+	struct vfio_ap_queue *q;
+
+	for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) {
+		q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi));
+
+		if (q)
+			vfio_ap_mdev_unlink_queue(q);
+	}
+}
+
 /**
  * unassign_adapter_store - parses the APID from @buf and clears the
  * corresponding bit in the mediated matrix device's APM
@@ -702,6 +773,7 @@ static ssize_t unassign_adapter_store(struct device *dev,
 	}
 
 	clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm);
+	vfio_ap_mdev_unlink_adapter(matrix_mdev, apid);
 	ret = count;
 done:
 	mutex_unlock(&matrix_dev->lock);
@@ -729,6 +801,16 @@ vfio_ap_mdev_verify_queues_reserved_for_apqi(struct ap_matrix_mdev *matrix_mdev,
 	return 0;
 }
 
+static void vfio_ap_mdev_link_domain(struct ap_matrix_mdev *matrix_mdev,
+				      unsigned long apqi)
+{
+	unsigned long apid;
+
+	for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES)
+		vfio_ap_mdev_link_apqn(matrix_mdev,
+				       AP_MKQID(apid, apqi));
+}
+
 /**
  * assign_domain_store - parses the APQI from @buf and sets the
  * corresponding bit in the mediated matrix device's AQM
@@ -794,6 +876,7 @@ static ssize_t assign_domain_store(struct device *dev,
 	if (ret)
 		goto share_err;
 
+	vfio_ap_mdev_link_domain(matrix_mdev, apqi);
 	ret = count;
 	goto done;
 
@@ -806,6 +889,19 @@ static ssize_t assign_domain_store(struct device *dev,
 }
 static DEVICE_ATTR_WO(assign_domain);
 
+static void vfio_ap_mdev_unlink_domain(struct ap_matrix_mdev *matrix_mdev,
+				       unsigned long apqi)
+{
+	unsigned long apid;
+	struct vfio_ap_queue *q;
+
+	for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) {
+		q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi));
+
+		if (q)
+			vfio_ap_mdev_unlink_queue(q);
+	}
+}
 
 /**
  * unassign_domain_store - parses the APQI from @buf and clears the
@@ -848,6 +944,7 @@ static ssize_t unassign_domain_store(struct device *dev,
 	}
 
 	clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm);
+	vfio_ap_mdev_unlink_domain(matrix_mdev, apqi);
 	ret = count;
 
 done:
@@ -1253,25 +1350,18 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, unsigned int retry)
 
 static int vfio_ap_mdev_reset_queues(struct ap_matrix_mdev *matrix_mdev)
 {
-	int ret;
-	int rc = 0;
-	unsigned long apid, apqi;
+	int ret, bkt, rc = 0;
 	struct vfio_ap_queue *q;
 
-	for_each_set_bit_inv(apid, matrix_mdev->matrix.apm,
-			     matrix_mdev->matrix.apm_max + 1) {
-		for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm,
-				     matrix_mdev->matrix.aqm_max + 1) {
-			q = vfio_ap_find_queue(AP_MKQID(apid, apqi));
-			ret = vfio_ap_mdev_reset_queue(q, 1);
-			/*
-			 * Regardless whether a queue turns out to be busy, or
-			 * is not operational, we need to continue resetting
-			 * the remaining queues.
-			 */
-			if (ret)
-				rc = ret;
-		}
+	hash_for_each(matrix_mdev->qtable.queues, bkt, q, mdev_qnode) {
+		ret = vfio_ap_mdev_reset_queue(q, 1);
+		/*
+		 * Regardless whether a queue turns out to be busy, or
+		 * is not operational, we need to continue resetting
+		 * the remaining queues.
+		 */
+		if (ret)
+			rc = ret;
 	}
 
 	return rc;
@@ -1411,6 +1501,28 @@ void vfio_ap_mdev_unregister(void)
 	mdev_unregister_driver(&vfio_ap_matrix_driver);
 }
 
+/*
+ * vfio_ap_queue_link_mdev
+ *
+ * @q: The queue to link with the matrix mdev.
+ *
+ * Links @q with the matrix mdev to which the queue's APQN is assigned.
+ */
+static void vfio_ap_queue_link_mdev(struct vfio_ap_queue *q)
+{
+	unsigned long apid = AP_QID_CARD(q->apqn);
+	unsigned long apqi = AP_QID_QUEUE(q->apqn);
+	struct ap_matrix_mdev *matrix_mdev;
+
+	list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) {
+		if (test_bit_inv(apid, matrix_mdev->matrix.apm) &&
+		    test_bit_inv(apqi, matrix_mdev->matrix.aqm)) {
+			vfio_ap_mdev_link_queue(matrix_mdev, q);
+			break;
+		}
+	}
+}
+
 int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
 {
 	struct vfio_ap_queue *q;
@@ -1418,9 +1530,11 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
 	q = kzalloc(sizeof(*q), GFP_KERNEL);
 	if (!q)
 		return -ENOMEM;
+
 	mutex_lock(&matrix_dev->lock);
 	q->apqn = to_ap_queue(&apdev->device)->qid;
 	q->saved_isc = VFIO_AP_ISC_INVALID;
+	vfio_ap_queue_link_mdev(q);
 	dev_set_drvdata(&apdev->device, q);
 	mutex_unlock(&matrix_dev->lock);
 
@@ -1433,6 +1547,10 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev)
 
 	mutex_lock(&matrix_dev->lock);
 	q = dev_get_drvdata(&apdev->device);
+
+	if (q->matrix_mdev)
+		vfio_ap_unlink_queue_fr_mdev(q);
+
 	vfio_ap_mdev_reset_queue(q, 1);
 	dev_set_drvdata(&apdev->device, NULL);
 	kfree(q);
drivers/s390/crypto/vfio_ap_private.h +14 -0
index 3cade25a1620..aea6a8b854b3 100644
@@ -19,6 +19,7 @@
 #include <linux/mutex.h>
 #include <linux/kvm_host.h>
 #include <linux/vfio.h>
+#include <linux/hashtable.h>
 
 #include "ap_bus.h"
 
@@ -74,6 +75,15 @@ struct ap_matrix {
 	DECLARE_BITMAP(adm, 256);
 };
 
+/**
+ * struct ap_queue_table - a table of queue objects.
+ *
+ * @queues: a hashtable of queues (struct vfio_ap_queue).
+ */
+struct ap_queue_table {
+	DECLARE_HASHTABLE(queues, 8);
+};
+
 /**
  * struct ap_matrix_mdev - Contains the data associated with a matrix mediated
  *			   device.
@@ -89,6 +99,7 @@ struct ap_matrix {
  * @pqap_hook:	the function pointer to the interception handler for the
  *		PQAP(AQIC) instruction.
  * @mdev:	the mediated device
+ * @qtable:	table of queues (struct vfio_ap_queue) assigned to the mdev
  */
 struct ap_matrix_mdev {
 	struct vfio_device vdev;
@@ -99,6 +110,7 @@ struct ap_matrix_mdev {
 	struct kvm *kvm;
 	crypto_hook pqap_hook;
 	struct mdev_device *mdev;
+	struct ap_queue_table qtable;
 };
 
 /**
@@ -108,6 +120,7 @@ struct ap_matrix_mdev {
  * @saved_pfn: the guest PFN pinned for the guest
  * @apqn: the APQN of the AP queue device
  * @saved_isc: the guest ISC registered with the GIB interface
+ * @mdev_qnode: allows the vfio_ap_queue struct to be added to a hashtable
  */
 struct vfio_ap_queue {
 	struct ap_matrix_mdev *matrix_mdev;
@@ -115,6 +128,7 @@ struct vfio_ap_queue {
 	int	apqn;
 #define VFIO_AP_ISC_INVALID 0xff
 	unsigned char saved_isc;
+	struct hlist_node mdev_qnode;
 };
 
 int vfio_ap_mdev_register(void);
Signature · 1 line
2.31.1
author maintainer
#21 lore raw

PATCH v18 9/18 s390/vfio-ap: introduce new mutex to control access to the KVM pointer

In reply to Tony Krowiak

The vfio_ap device driver registers for notification when the pointer to the KVM object for a guest is set. Recall that the KVM lock (kvm->lock) mutex must be taken outside of the matrix_dev->lock mutex to prevent the reporting by lockdep of a circular locking dependency (a.k.a., a lockdep splat):

* see commit 0cc00c8d4050 ("Fix circular lockdep when setting/clearing crypto masks")

* see commit 86956e70761b ("replace open coded locks for VFIO_GROUP_NOTIFY_SET_KVM notification")

With the introduction of support for hot plugging/unplugging AP devices passed through to a KVM guest, a new guests_lock mutex is introduced to ensure the proper locking order is maintained:

struct ap_matrix_dev {
        ...
        struct mutex guests_lock;
       ...
}

The matrix_dev->guests_lock controls access to the matrix_mdev instances that hold the state for AP devices that have been passed through to a KVM guest. This lock must be held:

1. To control access to the KVM pointer (matrix_mdev->kvm) while the vfio_ap device driver is using it to plug/unplug AP devices passed through to the KVM guest.

2. To add matrix_mdev instances to or remove them from matrix_dev->mdev_list. This is necessary to ensure the proper locking order when the list is iterated to find an ap_matrix_mdev instance for the purpose of plugging/unplugging AP devices passed through to a KVM guest; for example, when a queue device is bound to (probe callback) or removed from (remove callback) the vfio_ap device driver.

For example, when a queue device is removed from the vfio_ap device driver, if the adapter is passed through to a KVM guest, it will have to be unplugged. Since the vfio_ap device driver only knows the APQN of the queue device when it is removed, in order to figure out whether the adapter is passed through, the matrix_mdev object to which the queue is assigned will have to be found. Its KVM pointer (matrix_mdev->kvm) can then be used to determine if the mediated device is passed through (matrix_mdev->kvm != NULL) and if so, to unplug the adapter.

In addition to introducing the matrix_mdev->guests_lock mutex, the matrix_dev->lock mutex is being renamed to matrix_dev->mdevs_lock to better reflect its purpose, which is to control access to the state of the mediated devices under the control of the vfio_ap device driver.

It is not necessary to take the matrix_dev->guests_lock to access the KVM pointer if the pointer is not used to plug/unplug AP devices; however, in this case, the matrix_dev->mdevs_lock must be held in order to access the KVM pointer since it set and cleared under the protection of the mdevs_lock.
A case in point is the function that handles interception of the PQAP(AQIC) instruction sub-function by the host. This handler needs to access the KVM pointer, but only for the purposes of setting or clearing IRQ resources, so only the matrix_dev->mdevs_lock needs to be held.

The proper locking order must be maintained whenever plugging/unplugging AP devices passed through to a KVM guest under the control of the vfio_ap device driver:

    matrix_dev->guests_lock
    matrix_mdev->kvm->lock
    matrix_dev->mdevs_lock
---
 drivers/s390/crypto/vfio_ap_drv.c     |  7 ++-
 drivers/s390/crypto/vfio_ap_ops.c     | 81 +++++++++++++++------------
 drivers/s390/crypto/vfio_ap_private.h | 17 ++++--
 3 files changed, 61 insertions(+), 44 deletions(-)
drivers/s390/crypto/vfio_ap_drv.c +4 -3
index 5013a5531171..e8f3540ebfda 100644
@@ -68,7 +68,7 @@ static ssize_t status_show(struct device *dev,
 	struct ap_matrix_mdev *matrix_mdev;
 	struct ap_device *apdev = to_ap_dev(dev);
 
-	mutex_lock(&matrix_dev->lock);
+	mutex_lock(&matrix_dev->guests_lock);
 	q = dev_get_drvdata(&apdev->device);
 	matrix_mdev = vfio_ap_mdev_for_queue(q);
 
@@ -84,7 +84,7 @@ static ssize_t status_show(struct device *dev,
 				   AP_QUEUE_UNASSIGNED);
 	}
 
-	mutex_unlock(&matrix_dev->lock);
+	mutex_unlock(&matrix_dev->guests_lock);
 
 	return nchars;
 }
@@ -155,8 +155,9 @@ static int vfio_ap_matrix_dev_create(void)
 			goto matrix_alloc_err;
 	}
 
-	mutex_init(&matrix_dev->lock);
+	mutex_init(&matrix_dev->mdevs_lock);
 	INIT_LIST_HEAD(&matrix_dev->mdev_list);
+	mutex_init(&matrix_dev->guests_lock);
 
 	dev_set_name(&matrix_dev->device, "%s", VFIO_AP_DEV_NAME);
 	matrix_dev->device.parent = root_device;
drivers/s390/crypto/vfio_ap_ops.c +46 -35

277 lines not on this page · raw message

drivers/s390/crypto/vfio_ap_private.h +11 -6
index fa11a7e91e24..7e82a72d2083 100644
@@ -33,20 +33,25 @@
  * @available_instances: number of mediated matrix devices that can be created
  * @info:	the struct containing the output from the PQAP(QCI) instruction
  * @mdev_list:	the list of mediated matrix devices created
- * @lock:	mutex for locking the AP matrix device. This lock will be
- *		taken every time we fiddle with state managed by the vfio_ap
- *		driver, be it using @mdev_list or writing the state of a
- *		single ap_matrix_mdev device. It's quite coarse but we don't
- *		expect much contention.
+ * @mdevs_lock:	mutex for locking the ap_matrix_mdev devices under the control
+ *		of the vfio_ap device driver. This lock will be taken every time
+ *		we fiddle with state of an ap_matrix_mdev device. It's quite
+ *		coarse but we don't expect much contention.
  * @vfio_ap_drv: the vfio_ap device driver
+ * @guests_lock: mutex for controlling access to a guest that is using AP
+ *		 devices passed through by the vfio_ap device driver. This lock
+ *		 will be taken when the AP devices are plugged into or unplugged
+ *		 from a guest, and when an ap_matrix_mdev device is added to or
+ *		 removed from @mdev_list or the list is iterated.
  */
 struct ap_matrix_dev {
 	struct device device;
 	atomic_t available_instances;
 	struct ap_config_info info;
 	struct list_head mdev_list;
-	struct mutex lock;
+	struct mutex mdevs_lock;
 	struct ap_driver  *vfio_ap_drv;
+	struct mutex guests_lock;
 };
 
 extern struct ap_matrix_dev *matrix_dev;
Signature · 1 line
2.31.1
author maintainer
#10 lore raw

PATCH v18 8/18 s390/vfio-ap: allow assignment of unavailable AP queues to mdev device

In reply to Tony Krowiak

The current implementation does not allow assignment of an AP adapter or domain to an mdev device if each APQN resulting from the assignment does not reference an AP queue device that is bound to the vfio_ap device driver. This patch allows assignment of AP resources to the matrix mdev as long as the APQNs resulting from the assignment:
1. Are not reserved by the AP BUS for use by the zcrypt device drivers.
2. Are not assigned to another matrix mdev.

The rationale behind this is that the AP architecture does not preclude assignment of APQNs to an AP configuration profile that are not available to the system.

---
 drivers/s390/crypto/vfio_ap_ops.c | 224 +++++++-----------------------
 1 file changed, 53 insertions(+), 171 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +53 -171
index b67b2f0faeea..f2b98f347f9f 100644
@@ -525,141 +525,48 @@ static struct attribute_group *vfio_ap_mdev_type_groups[] = {
 	NULL,
 };
 
-struct vfio_ap_queue_reserved {
-	unsigned long *apid;
-	unsigned long *apqi;
-	bool reserved;
-};
+#define MDEV_SHARING_ERR "Userspace may not re-assign queue %02lx.%04lx " \
+			 "already assigned to %s"
 
-/**
- * vfio_ap_has_queue - determines if the AP queue containing the target in @data
- *
- * @dev: an AP queue device
- * @data: a struct vfio_ap_queue_reserved reference
- *
- * Flags whether the AP queue device (@dev) has a queue ID containing the APQN,
- * apid or apqi specified in @data:
- *
- * - If @data contains both an apid and apqi value, then @data will be flagged
- *   as reserved if the APID and APQI fields for the AP queue device matches
- *
- * - If @data contains only an apid value, @data will be flagged as
- *   reserved if the APID field in the AP queue device matches
- *
- * - If @data contains only an apqi value, @data will be flagged as
- *   reserved if the APQI field in the AP queue device matches
- *
- * Return: 0 to indicate the input to function succeeded. Returns -EINVAL if
- * @data does not contain either an apid or apqi.
- */
-static int vfio_ap_has_queue(struct device *dev, void *data)
+static void vfio_ap_mdev_log_sharing_err(struct ap_matrix_mdev *matrix_mdev,
+					 unsigned long *apm,
+					 unsigned long *aqm)
 {
-	struct vfio_ap_queue_reserved *qres = data;
-	struct ap_queue *ap_queue = to_ap_queue(dev);
-	ap_qid_t qid;
-	unsigned long id;
-
-	if (qres->apid && qres->apqi) {
-		qid = AP_MKQID(*qres->apid, *qres->apqi);
-		if (qid == ap_queue->qid)
-			qres->reserved = true;
-	} else if (qres->apid && !qres->apqi) {
-		id = AP_QID_CARD(ap_queue->qid);
-		if (id == *qres->apid)
-			qres->reserved = true;
-	} else if (!qres->apid && qres->apqi) {
-		id = AP_QID_QUEUE(ap_queue->qid);
-		if (id == *qres->apqi)
-			qres->reserved = true;
-	} else {
-		return -EINVAL;
-	}
-
-	return 0;
-}
-
-/**
- * vfio_ap_verify_queue_reserved - verifies that the AP queue containing
- * @apid or @aqpi is reserved
- *
- * @apid: an AP adapter ID
- * @apqi: an AP queue index
- *
- * Verifies that the AP queue with @apid/@apqi is reserved by the VFIO AP device
- * driver according to the following rules:
- *
- * - If both @apid and @apqi are not NULL, then there must be an AP queue
- *   device bound to the vfio_ap driver with the APQN identified by @apid and
- *   @apqi
- *
- * - If only @apid is not NULL, then there must be an AP queue device bound
- *   to the vfio_ap driver with an APQN containing @apid
- *
- * - If only @apqi is not NULL, then there must be an AP queue device bound
- *   to the vfio_ap driver with an APQN containing @apqi
- *
- * Return: 0 if the AP queue is reserved; otherwise, returns -EADDRNOTAVAIL.
- */
-static int vfio_ap_verify_queue_reserved(unsigned long *apid,
-					 unsigned long *apqi)
-{
-	int ret;
-	struct vfio_ap_queue_reserved qres;
-
-	qres.apid = apid;
-	qres.apqi = apqi;
-	qres.reserved = false;
-
-	ret = driver_for_each_device(&matrix_dev->vfio_ap_drv->driver, NULL,
-				     &qres, vfio_ap_has_queue);
-	if (ret)
-		return ret;
-
-	if (qres.reserved)
-		return 0;
-
-	return -EADDRNOTAVAIL;
-}
-
-static int
-vfio_ap_mdev_verify_queues_reserved_for_apid(struct ap_matrix_mdev *matrix_mdev,
-					     unsigned long apid)
-{
-	int ret;
-	unsigned long apqi;
-	unsigned long nbits = matrix_mdev->matrix.aqm_max + 1;
-
-	if (find_first_bit_inv(matrix_mdev->matrix.aqm, nbits) >= nbits)
-		return vfio_ap_verify_queue_reserved(&apid, NULL);
-
-	for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, nbits) {
-		ret = vfio_ap_verify_queue_reserved(&apid, &apqi);
-		if (ret)
-			return ret;
-	}
+	unsigned long apid, apqi;
+	const struct device *dev = mdev_dev(matrix_mdev->mdev);
+	const char *mdev_name = dev_name(dev);
 
-	return 0;
+	for_each_set_bit_inv(apid, apm, AP_DEVICES)
+		for_each_set_bit_inv(apqi, aqm, AP_DOMAINS)
+			dev_warn(dev, MDEV_SHARING_ERR, apid, apqi, mdev_name);
 }
 
 /**
- * vfio_ap_mdev_verify_no_sharing - verifies that the AP matrix is not configured
+ * vfio_ap_mdev_verify_no_sharing - verify APQNs are not shared by matrix mdevs
  *
- * @matrix_mdev: the mediated matrix device
+ * @mdev_apm: mask indicating the APIDs of the APQNs to be verified
+ * @mdev_aqm: mask indicating the APQIs of the APQNs to be verified
  *
- * Verifies that the APQNs derived from the cross product of the AP adapter IDs
- * and AP queue indexes comprising the AP matrix are not configured for another
+ * Verifies that each APQN derived from the Cartesian product of a bitmap of
+ * AP adapter IDs and AP queue indexes is not configured for any matrix
  * mediated device. AP queue sharing is not allowed.
  *
- * Return: 0 if the APQNs are not shared; otherwise returns -EADDRINUSE.
+ * Return: 0 if the APQNs are not shared; otherwise return -EADDRINUSE.
  */
-static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
+static int vfio_ap_mdev_verify_no_sharing(unsigned long *mdev_apm,
+					  unsigned long *mdev_aqm)
 {
-	struct ap_matrix_mdev *lstdev;
+	struct ap_matrix_mdev *matrix_mdev;
 	DECLARE_BITMAP(apm, AP_DEVICES);
 	DECLARE_BITMAP(aqm, AP_DOMAINS);
 
-	list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) {
-		if (matrix_mdev == lstdev)
+	list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) {
+		/*
+		 * If the input apm and aqm belong to the matrix_mdev's matrix,
+		 * then move on to the next.
+		 */
+		if (mdev_apm == matrix_mdev->matrix.apm &&
+		    mdev_aqm == matrix_mdev->matrix.aqm)
 			continue;
 
 		memset(apm, 0, sizeof(apm));
@@ -669,20 +576,32 @@ static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
 		 * We work on full longs, as we can only exclude the leftover
 		 * bits in non-inverse order. The leftover is all zeros.
 		 */
-		if (!bitmap_and(apm, matrix_mdev->matrix.apm,
-				lstdev->matrix.apm, AP_DEVICES))
+		if (!bitmap_and(apm, mdev_apm, matrix_mdev->matrix.apm,
+				AP_DEVICES))
 			continue;
 
-		if (!bitmap_and(aqm, matrix_mdev->matrix.aqm,
-				lstdev->matrix.aqm, AP_DOMAINS))
+		if (!bitmap_and(aqm, mdev_aqm, matrix_mdev->matrix.aqm,
+				AP_DOMAINS))
 			continue;
 
+		vfio_ap_mdev_log_sharing_err(matrix_mdev, apm, aqm);
+
 		return -EADDRINUSE;
 	}
 
 	return 0;
 }
 
+static int vfio_ap_mdev_validate_masks(struct ap_matrix_mdev *matrix_mdev)
+{
+	if (ap_apqn_in_matrix_owned_by_def_drv(matrix_mdev->matrix.apm,
+					       matrix_mdev->matrix.aqm))
+		return -EADDRNOTAVAIL;
+
+	return vfio_ap_mdev_verify_no_sharing(matrix_mdev->matrix.apm,
+					      matrix_mdev->matrix.aqm);
+}
+
 static void vfio_ap_mdev_link_adapter(struct ap_matrix_mdev *matrix_mdev,
 				      unsigned long apid)
 {
@@ -750,30 +669,19 @@ static ssize_t assign_adapter_store(struct device *dev,
 		goto done;
 	}
 
-	/*
-	 * Set the bit in the AP mask (APM) corresponding to the AP adapter
-	 * number (APID). The bits in the mask, from most significant to least
-	 * significant bit, correspond to APIDs 0-255.
-	 */
-	ret = vfio_ap_mdev_verify_queues_reserved_for_apid(matrix_mdev, apid);
-	if (ret)
-		goto done;
-
 	set_bit_inv(apid, matrix_mdev->matrix.apm);
 	memset(apm, 0, sizeof(apm));
 	set_bit_inv(apid, apm);
 
-	ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev);
-	if (ret)
-		goto share_err;
+	ret = vfio_ap_mdev_validate_masks(matrix_mdev);
+	if (ret) {
+		clear_bit_inv(apid, matrix_mdev->matrix.apm);
+		goto done;
+	}
 
 	vfio_ap_mdev_link_adapter(matrix_mdev, apid);
 	vfio_ap_mdev_filter_matrix(apm, matrix_mdev->matrix.aqm, matrix_mdev);
 	ret = count;
-	goto done;
-
-share_err:
-	clear_bit_inv(apid, matrix_mdev->matrix.apm);
 done:
 	mutex_unlock(&matrix_dev->lock);
 
@@ -848,26 +756,6 @@ static ssize_t unassign_adapter_store(struct device *dev,
 }
 static DEVICE_ATTR_WO(unassign_adapter);
 
-static int
-vfio_ap_mdev_verify_queues_reserved_for_apqi(struct ap_matrix_mdev *matrix_mdev,
-					     unsigned long apqi)
-{
-	int ret;
-	unsigned long apid;
-	unsigned long nbits = matrix_mdev->matrix.apm_max + 1;
-
-	if (find_first_bit_inv(matrix_mdev->matrix.apm, nbits) >= nbits)
-		return vfio_ap_verify_queue_reserved(NULL, &apqi);
-
-	for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, nbits) {
-		ret = vfio_ap_verify_queue_reserved(&apid, &apqi);
-		if (ret)
-			return ret;
-	}
-
-	return 0;
-}
-
 static void vfio_ap_mdev_link_domain(struct ap_matrix_mdev *matrix_mdev,
 				      unsigned long apqi)
 {
@@ -934,25 +822,19 @@ static ssize_t assign_domain_store(struct device *dev,
 		goto done;
 	}
 
-	ret = vfio_ap_mdev_verify_queues_reserved_for_apqi(matrix_mdev, apqi);
-	if (ret)
-		goto done;
-
 	set_bit_inv(apqi, matrix_mdev->matrix.aqm);
 	memset(aqm, 0, sizeof(aqm));
 	set_bit_inv(apqi, aqm);
 
-	ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev);
-	if (ret)
-		goto share_err;
+	ret = vfio_ap_mdev_validate_masks(matrix_mdev);
+	if (ret) {
+		clear_bit_inv(apqi, matrix_mdev->matrix.aqm);
+		goto done;
+	}
 
 	vfio_ap_mdev_link_domain(matrix_mdev, apqi);
 	vfio_ap_mdev_filter_matrix(matrix_mdev->matrix.apm, aqm, matrix_mdev);
 	ret = count;
-	goto done;
-
-share_err:
-	clear_bit_inv(apqi, matrix_mdev->matrix.aqm);
 done:
 	mutex_unlock(&matrix_dev->lock);
 
Signature · 1 line
2.31.1
author maintainer
#32 lore raw

PATCH v18 13/18 s390/vfio-ap: implement in-use callback for vfio_ap driver

In reply to Tony Krowiak

Let's implement the callback to indicate when an APQN
is in use by the vfio_ap device driver. The callback is invoked whenever a change to the apmask or aqmask would result in one or more queue devices being removed from the driver. The vfio_ap device driver will indicate a resource is in use
if the APQN of any of the queue devices to be removed are assigned to any of the matrix mdevs under the driver's control.

There is potential for a deadlock condition between the matrix_dev->lock used to lock the matrix device during assignment of adapters and domains and the ap_perms_mutex locked by the AP bus when changes are made to the sysfs apmask/aqmask attributes.

The AP Perms lock controls access to the objects that store the adapter numbers (ap_perms) and domain numbers (aq_perms) for the sysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attributes. These attributes identify which queues are reserved for the zcrypt default device drivers. Before allowing a bit to be removed from either mask, the AP bus must check with the vfio_ap device driver to verify that none of the queues are assigned to any of its mediated devices.

The apmask/aqmask attributes can be written or read at any time from userspace, so care must be taken to prevent a deadlock with asynchronous operations that might be taking place in the vfio_ap device driver. For example, consider the following:

1. A system administrator assigns an adapter to a mediated device under the control of the vfio_ap device driver. The driver will need to first take the matrix_dev->guests_lock to potentially hot plug the adapter into the KVM guest.
2. At the same time, a system administrator sets a bit in the sysfs /sys/bus/ap/ap_mask attribute. To complete the operation, the AP bus must:
a. Take the ap_perms_mutex lock to update the object storing the values
for the /sys/bus/ap/ap_mask attribute.
b. Call the vfio_ap device driver's in-use callback to verify that the
queues now being reserved for the default zcrypt drivers are not assigned to a mediated device owned by the vfio_ap device driver. To do the verification, the in-use callback function takes the matrix_dev->guests_lock, but has to wait because it is already held by the operation in 1 above.
3. The vfio_ap device driver calls an AP bus function to verify that the new queues resulting from the assignment of the adapter in step 1 are not reserved for the default zcrypt device driver. This AP bus function tries to take the ap_perms_mutex lock but gets stuck waiting for the waiting for the lock due to step 2a above.

Consequently, we have the following deadlock situation:

matrix_dev->lock locked (1)
ap_perms_mutex lock locked (2a)
Waiting for matrix_dev->lock (2b) which is currently held (1) Waiting for ap_perms_mutex lock (3) which is currently held (2a)

To prevent this deadlock scenario, the in-use callback will use mutex_trylock to take the matrix_dev->guests_lock and if the lock can not be obtained, will return -EBUSY from the function.

---
 drivers/s390/crypto/vfio_ap_drv.c     |  1 +
 drivers/s390/crypto/vfio_ap_ops.c     | 81 ++++++++++++++++++++++++++-
 drivers/s390/crypto/vfio_ap_private.h |  2 +
 3 files changed, 82 insertions(+), 2 deletions(-)
drivers/s390/crypto/vfio_ap_drv.c +1 -0
index e8f3540ebfda..c4f0e28b1e91 100644
@@ -103,6 +103,7 @@ static const struct attribute_group vfio_queue_attr_group = {
 static struct ap_driver vfio_ap_drv = {
 	.probe = vfio_ap_mdev_probe_queue,
 	.remove = vfio_ap_mdev_remove_queue,
+	.in_use = vfio_ap_mdev_resource_in_use,
 	.ids = ap_queue_ids,
 };
 
drivers/s390/crypto/vfio_ap_ops.c +79 -2

104 lines not on this page · raw message

drivers/s390/crypto/vfio_ap_private.h +2 -0
index a3811cd9852d..e30f171f89ec 100644
@@ -146,4 +146,6 @@ void vfio_ap_mdev_unregister(void);
 int vfio_ap_mdev_probe_queue(struct ap_device *queue);
 void vfio_ap_mdev_remove_queue(struct ap_device *queue);
 
+int vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm);
+
 #endif /* _VFIO_AP_PRIVATE_H_ */
Signature · 1 line
2.31.1
author maintainer
#27 lore raw

PATCH v18 11/18 s390/vfio-ap: hot plug/unplug of AP devices when probed/removed

In reply to Tony Krowiak

When an AP queue device is probed or removed, if the mediated device is passed through to a KVM guest, the mediated device's adapter, domain and control domain bitmaps must be filtered to update the guest's APCB and if any changes are detected, the guest's APCB must then be hot plugged into the guest to reflect those changes to the guest.

The following locks must be taken in order to ensure the operation can be validly executed and a lockdep splat prevented:

matrix_dev->guests_lock => matrix_mdev->kvm->lock => matrix_dev->mdevs_lock

Note: The matrix_mdev->kvm->lock can only be taken if the mediated device

      is passed through to a guest (i.e., matrix_mdev->kvm is not NULL).
---
 drivers/s390/crypto/vfio_ap_ops.c | 136 +++++++++++++++++++++++++-----
 1 file changed, 113 insertions(+), 23 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +113 -23
index 4c382cd3afc7..f0fdfbbe2d29 100644
@@ -1523,73 +1523,163 @@ void vfio_ap_mdev_unregister(void)
 	mdev_unregister_driver(&vfio_ap_matrix_driver);
 }
 
-/*
- * vfio_ap_queue_link_mdev
+
+
+/**
+ * vfio_ap_mdev_get_qlocks_4_probe: acquire all of the locks required to probe
+ *				    a queue device.
  *
- * @q: The queue to link with the matrix mdev.
+ * @apqn: the APQN of the queue device being probed
  *
- * Links @q with the matrix mdev to which the queue's APQN is assigned.
+ * Return: the matrix mdev to which @apqn is assigned.
  */
-static void vfio_ap_queue_link_mdev(struct vfio_ap_queue *q)
+static struct ap_matrix_mdev *vfio_ap_mdev_get_qlocks_4_probe(int apqn)
 {
-	unsigned long apid = AP_QID_CARD(q->apqn);
-	unsigned long apqi = AP_QID_QUEUE(q->apqn);
 	struct ap_matrix_mdev *matrix_mdev;
+	unsigned long apid = AP_QID_CARD(apqn);
+	unsigned long apqi = AP_QID_QUEUE(apqn);
+
+	/*
+	 * Lock the mutex required to access the list of mdevs under the control
+	 * of the vfio_ap device driver and access the KVM guest's state
+	 */
+	mutex_lock(&matrix_dev->guests_lock);
 
 	list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) {
 		if (test_bit_inv(apid, matrix_mdev->matrix.apm) &&
 		    test_bit_inv(apqi, matrix_mdev->matrix.aqm)) {
-			vfio_ap_mdev_link_queue(matrix_mdev, q);
-			break;
+			/*
+			 * If the KVM guest is running, lock the mutex required
+			 * to plug/unplug AP devices passed through to the
+			 * guest.
+			 */
+			if (matrix_mdev->kvm)
+				mutex_lock(&matrix_mdev->kvm->lock);
+
+			/*
+			 * Lock the mutex required to access the mdev's state.
+			 */
+			mutex_lock(&matrix_dev->mdevs_lock);
+
+			return matrix_mdev;
 		}
 	}
+
+	return NULL;
+}
+
+/**
+ * vfio_ap_mdev_put_qlocks - unlock all of the locks acquired for probing or
+ *			     removing a queue device.
+ *
+ * @matrix_mdev: the mdev to which the queue being probed/removed is assigned.
+ */
+static void vfio_ap_mdev_put_qlocks(struct ap_matrix_mdev *matrix_mdev)
+{
+	if (matrix_mdev) {
+		/*
+		 * Unlock the queue required for accessing the state of
+		 * matrix_mdev
+		 */
+		mutex_unlock(&matrix_dev->mdevs_lock);
+
+		/*
+		 * If a KVM guest is currently running, unlock the mutex
+		 * required to plug/unplug AP devices passed through to the
+		 * guest.
+		 */
+		if (matrix_mdev && matrix_mdev->kvm)
+			mutex_unlock(&matrix_mdev->kvm->lock);
+	}
+
+	/* Unlock the mutex required to access the KVM guest's state */
+	mutex_unlock(&matrix_dev->guests_lock);
 }
 
 int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
 {
 	struct vfio_ap_queue *q;
 	DECLARE_BITMAP(apm, AP_DEVICES);
+	struct ap_matrix_mdev *matrix_mdev;
 
 	q = kzalloc(sizeof(*q), GFP_KERNEL);
 	if (!q)
 		return -ENOMEM;
 
-	mutex_lock(&matrix_dev->guests_lock);
-	mutex_lock(&matrix_dev->mdevs_lock);
 	q->apqn = to_ap_queue(&apdev->device)->qid;
 	q->saved_isc = VFIO_AP_ISC_INVALID;
-	vfio_ap_queue_link_mdev(q);
-	if (q->matrix_mdev) {
+	matrix_mdev = vfio_ap_mdev_get_qlocks_4_probe(q->apqn);
+	if (matrix_mdev) {
+		vfio_ap_mdev_link_queue(matrix_mdev, q);
 		memset(apm, 0, sizeof(apm));
 		set_bit_inv(AP_QID_CARD(q->apqn), apm);
-		vfio_ap_mdev_filter_matrix(apm, q->matrix_mdev->matrix.aqm,
-					   q->matrix_mdev);
+		if (vfio_ap_mdev_filter_matrix(apm, q->matrix_mdev->matrix.aqm,
+					       q->matrix_mdev))
+			vfio_ap_mdev_hotplug_apcb(q->matrix_mdev);
 	}
 	dev_set_drvdata(&apdev->device, q);
-	mutex_unlock(&matrix_dev->mdevs_lock);
-	mutex_unlock(&matrix_dev->guests_lock);
+	vfio_ap_mdev_put_qlocks(matrix_mdev);
 
 	return 0;
 }
 
-void vfio_ap_mdev_remove_queue(struct ap_device *apdev)
+/**
+ * vfio_ap_get_qlocks_4_rem: acquire all of the locks required to remove a
+ *			     queue device.
+ *
+ * @matrix_mdev: the device to which the APQN of the queue device being removed is
+ *		 assigned.
+ */
+static struct vfio_ap_queue *vfio_ap_get_qlocks_4_rem(struct ap_device *apdev)
 {
-	unsigned long apid;
 	struct vfio_ap_queue *q;
 
-	mutex_lock(&matrix_dev->mdevs_lock);
+	/* Lock the mutex required to access the KVM guest's state */
+	mutex_lock(&matrix_dev->guests_lock);
+
 	q = dev_get_drvdata(&apdev->device);
 
+	/*
+	 * If the queue is assigned to a mediated device and a KVM guest is
+	 * currently running, lock the mutex required to plug/unplug AP devices
+	 * passed through to the guest.
+	 */
 	if (q->matrix_mdev) {
+		if (q->matrix_mdev->kvm)
+			mutex_lock(&q->matrix_mdev->kvm->lock);
+		/*
+		 * Lock the mutex required to access the state of the
+		 * matrix_mdev
+		 */
+		mutex_lock(&matrix_dev->mdevs_lock);
+	}
+
+	return q;
+}
+
+void vfio_ap_mdev_remove_queue(struct ap_device *apdev)
+{
+	unsigned long apid, apqi;
+	struct vfio_ap_queue *q;
+	struct ap_matrix_mdev *matrix_mdev;
+
+	q = vfio_ap_get_qlocks_4_rem(apdev);
+	matrix_mdev = q->matrix_mdev;
+
+	if (matrix_mdev) {
 		vfio_ap_unlink_queue_fr_mdev(q);
 
 		apid = AP_QID_CARD(q->apqn);
-		if (test_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm))
-			clear_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm);
+		apqi = AP_QID_QUEUE(q->apqn);
+		if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) &&
+		    test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) {
+			clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm);
+			vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+		}
 	}
 
 	vfio_ap_mdev_reset_queue(q, 1);
 	dev_set_drvdata(&apdev->device, NULL);
 	kfree(q);
-	mutex_unlock(&matrix_dev->mdevs_lock);
+	vfio_ap_mdev_put_qlocks(matrix_mdev);
 }
Signature · 1 line
2.31.1
author maintainer
#42 lore raw

PATCH v18 17/18 s390/Docs: new doc describing lock usage by the vfio_ap device driver

In reply to Tony Krowiak

Introduces a new document describing the locks used by the vfio_ap device driver and how to use them so as to avoid lockdep reports and deadlock situations.

---
 Documentation/s390/vfio-ap-locking.rst | 389 +++++++++++++++++++++++++
 1 file changed, 389 insertions(+)
 create mode 100644 Documentation/s390/vfio-ap-locking.rst
Documentation/s390/vfio-ap-locking.rst new file +389 -0
new file mode 100644
index 000000000000..10abbb6d6089
@@ -0,0 +1,389 @@
+======================
+VFIO AP Locks Overview
+======================
+This document describes the locks that are pertinent to the secure operation
+of the vfio_ap device driver. Throughout this document, the following variables
+will be used to denote instances of the structures herein described:
+
+struct ap_matrix_dev *matrix_dev;
+struct ap_matrix_mdev *matrix_mdev;
+struct kvm *kvm;
+
+The Matrix Devices Lock (drivers/s390/crypto/vfio_ap_private.h)
+--------------------------------------------------------------
+
+struct ap_matrix_dev {
+	...
+	struct list_head mdev_list;
+	struct mutex mdevs_lock;
+	...
+}
+
+The Matrix Devices Lock (matrix_dev->mdevs_lock) is implemented as a global
+mutex contained within the single instance of struct ap_matrix_dev. This lock
+controls access to all fields contained within each matrix_mdev instance under
+the control of the vfio_ap device driver (matrix_dev->mdev_list). This lock must
+be held while reading from, writing to or using the data from a field contained
+within a matrix_mdev instance representing one of the vfio_ap device driver's
+mediated devices.
+
+The KVM Lock (include/linux/kvm_host.h)
+---------------------------------------
+
+struct kvm {
+	...
+	struct mutex lock;
+	...
+}
+
+The KVM Lock (kvm->lock) controls access to the state data for a KVM guest. This
+lock must be held by the vfio_ap device driver while one or more AP adapters,
+domains or control domains are being plugged into or unplugged from the guest.
+
+The vfio_ap device driver registers a function to be notified when the pointer
+to the kvm instance has been set. The KVM pointer is passed to the handler by
+the notifier and is stored in the in the matrix_mdev instance
+(matrix_mdev->kvm = kvm) containing the state of the mediated device that has
+been passed through to the KVM guest.
+
+The Guests Lock (drivers/s390/crypto/vfio_ap_private.h)
+-----------------------------------------------------------
+
+struct ap_matrix_dev {
+	...
+	struct list_head mdev_list;
+	struct mutex guests_lock;
+	...
+}
+
+The Guests Lock (matrix_dev->guests_lock) controls access to the
+matrix_mdev instances (matrix_dev->mdev_list) that represent mediated devices
+that hold the state for the mediated devices that have been passed through to a
+KVM guest. This lock must be held:
+
+1. To control access to the KVM pointer (matrix_mdev->kvm) while the vfio_ap
+   device driver is using it to plug/unplug AP devices passed through to the KVM
+   guest.
+
+2. To add matrix_mdev instances to or remove them from matrix_dev->mdev_list.
+   This is necessary to ensure the proper locking order when the list is perused
+   to find an ap_matrix_mdev instance for the purpose of plugging/unplugging
+   AP devices passed through to a KVM guest.
+
+   For example, when a queue device is removed from the vfio_ap device driver,
+   if the adapter is passed through to a KVM guest, it will have to be
+   unplugged. In order to figure out whether the adapter is passed through,
+   the matrix_mdev object to which the queue is assigned will have to be
+   found. The KVM pointer (matrix_mdev->kvm) can then be used to determine if
+   the mediated device is passed through (matrix_mdev->kvm != NULL) and if so,
+   to unplug the adapter.
+
+It is not necessary to take the Guests Lock to access the KVM pointer if the
+pointer is not used to plug/unplug devices passed through to the KVM guest;
+however, in this case, the Matrix Devices Lock (matrix_dev->mdevs_lock) must be
+held in order to access the KVM pointer since it set and cleared under the
+protection of the Matrix Devices Lock. A case in point is the function that
+handles interception of the PQAP(AQIC) instruction sub-function. This handler
+needs to access the KVM pointer only for the purposes of setting or clearing IRQ
+resources, so only the matrix_dev->mdevs_lock needs to be held.
+
+The PQAP Hook Lock (arch/s390/include/asm/kvm_host.h)
+-----------------------------------------------------
+
+typedef int (*crypto_hook)(struct kvm_vcpu *vcpu);
+
+struct kvm_s390_crypto {
+	...
+	struct rw_semaphore pqap_hook_rwsem;
+	crypto_hook *pqap_hook;
+	...
+};
+
+The PQAP Hook Lock is a r/w semaphore that controls access to the function
+pointer of the handler (*kvm->arch.crypto.pqap_hook) to invoke when the
+PQAP(AQIC) instruction sub-function is intercepted by the host. The lock must be
+held in write mode when pqap_hook value is set, and in read mode when the
+pqap_hook function is called.
+
+Locking Order
+-------------
+
+If the various locks are not taken in the proper order, it could potentially
+result in a lockdep splat. The proper order for taking locks depends upon
+the operation taking place, but in general the Guests Lock
+(matrix_dev->guests_lock) must be taken outside of the KVM Lock (kvm->lock)
+which in turn must be taken outside of the Matrix Devices Lock
+(matrix_dev->mdevs_lock).
+
+The following describes the various operations under which the various locks are
+taken, the purpose for taking them and the order in which they must be taken.
+
+* Operations: Setting or clearing the KVM pointer (matrix_mdev->kvm):
+
+  1. PQAP Hook Lock (kvm->arch.crypto.pqap_hook_rwsem):
+
+	This semaphore must be held in write mode while setting or clearing the
+	reference to the function pointer (kvm->arch.crypt.pqap_hook) to call
+	when the PQAP(AQIC) instruction sub-function is intercepted by the host.
+	The function pointer is set when the KVM pointer is being set and
+	cleared when the KVM pointer is being cleared.
+
+  2.Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held while accessing the KVM pointer
+	(matrix_mdev->kvm) to plug/unplug AP devices passed through to the
+	KVM guest
+
+  3. KVM Lock (kvm->lock):
+
+	This mutex must be held while the AP devices passed through to the KVM
+	guest are plugged/unplugged.
+
+  4. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the operation.
+
+* Operations: Assign or unassign an adapter, domain or control domain of a
+	      mediated device under the control of the vfio_ap device driver:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held while accessing the KVM pointer
+	(matrix_dev->kvm) to plug/unplug AP devices passed through to the
+	KVM guest as a result of the assignment/unassignment operation.
+	Assignment of an AP device may result in additional queue devices
+	or control domains being plugged into the guest. Similarly, unassignment
+	may result in unplugging queue devices or control domains from the
+	guest
+
+  3. KVM Lock (matrix_mdev->kvm->lock):
+
+	This mutex must be held while the AP devices passed through to the KVM
+	guest are plugged in or unplugged.
+
+  4. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the operation. For example, to
+	determine which AP devices need to be plugged/unplugged, the lock
+	must be held to prevent other operations from changing the data used
+	to construct the guest's AP configuration.
+
+* Operations: Probe or remove an AP queue device:
+
+  When a queue device is bound to the vfio_ap device driver, the driver's probe
+  callback is invoked. Similarly, when a queue device is unbound from the
+  driver it's remove callback is invoked. The probe and remove functions will
+  take locks in the following order:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held for the duration of this operation.
+
+	At the time of the operation, the vfio_ap device driver will only have
+	the APQN of the queue being probed or removed, so the
+	matrix_dev->mdevs_list must be perused to locate the matrix_mdev
+	instance to which the queue is assigned. The Guests Lock must be held
+	during this time to prevent the list from being changed while processing
+	the probe/remove.
+
+	Once the matrix_mdev is found, the operation must determine whether the
+	mediated device is passed through to a guest (matrix_mdev->kvm != NULL),
+	then use the KVM pointer to perform the plug/unplug operation. Here
+	again, the lock must be held to prevent other operations from accessing
+	the KVM pointer for the same purpose.
+
+  2. KVM Lock (kvm->lock):
+
+	This mutex must be held while the AP devices passed through to the KVM
+	guest are plugged in or unplugged to prevent other operations from
+	accessing the guest's state while it is in flux.
+
+  3. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the operation, such as the
+	masks used to construct the KVM guest's AP configuration.
+
+* Operations: Probe or remove a mediated device:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held while adding the matrix_mdev to the
+	matrix_dev->mdev_list during the probe operation or when removing it
+	from the list during the remove operation. This is to prevent access by
+	other functions that must traverse the list to find a matrix_mdev for
+	the purpose of plugging/unplugging AP devices passed through to a KVM
+	guest (i.e., probe/remove queue callbacks), while the list is being
+	modified.
+
+  2. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the probe or remove operations
+	such as initializing the hashtable of queue devices
+	(matrix_mdev->qtable.queues) assigned to the matrix_mdev.
+
+* Operation: Handle interception of the PQAP(AQIC) instruction sub-function:
+
+  1. PQAP Hook Lock (kvm->arch.crypto.pqap_hook_rwsem)
+
+	This semaphore must be held in read mode while retrieving the function
+	pointer (kvm->arch.crypto.pqap_hook) and executing the function that
+	handles the interception of the PQAP(AQIC) instruction sub-function by
+	the host.
+
+  2. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the execution of the
+	PQAP(AQIC) instruction sub-function interception handler. For example,
+	the handler must iterate over the matrix_mdev->qtable.queues hashtable
+	to find the vfio_ap_queue object representing the queue for which
+	interrupts are being enabled or disabled.
+
+  Note: It is not necessary to take the Guests Lock (matrix_dev->guests_lock)
+	or the KVM Lock (matrix_mdev->kvm->lock) because the KVM pointer
+	will not be accessed to plug/unplug AP devices passed through to the
+	guest; it will only be used to allocate or free resources for processing
+	interrupts.
+
+* Operation: Handle AP configuration changed notification:
+
+  The vfio_ap device driver registers a callback function to be notified when
+  the AP bus detects that the host's AP configuration has changed. This can
+  occur due to the addition or removal of AP adapters, domains or control
+  domains via an SE or HMC connected to a DPM enabled LPAR. The objective of the
+  handler is to remove the queues no longer accessible via the host in bulk
+  rather than one queue at a time via the driver's queue device remove callback.
+  The locks and the order in which they must be taken by this operation are:
+
+ 1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held for the duration of the operation to:
+
+	* Iterate over the matrix_dev->mdev_list to find each matrix_mdev from
+	  which a queue device to be removed is assigned and prevent other
+	  operations from modifying the list while processing the affected
+	  matrix_mdev instances.
+
+	* Prevent other operations from acquiring access to the KVM pointer in
+	  each affected matrix_mdev instance (matrix_mdev->kvm) for the purpose
+	  of plugging/unplugging AP devices passed through to the KVM guest via
+	  that instance.
+
+2. KVM Lock (kvm->lock):
+
+	This mutex must be held for each affected matrix_mdev instance while
+	the AP devices passed through to the KVM guest are unplugged to prevent
+	other operations from accessing the guest's state while it is in flux.
+
+	Note: This lock must be re-acquired for each matrix_mdev instance.
+
+  3. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held for each affected matrix_mdev to prevent access
+	to the matrix_mdev state while writing/reading state values during the
+	operation, such as the masks used to construct the KVM guest's AP
+	configuration.
+
+	Note: This lock must be re-acquired for each matrix_mdev instance.
+
+Operation: Handle AP bus scan complete notification:
+
+  The vfio_ap device driver registers a callback function to be notified when
+  the AP bus scan completes after detecting the addition or removal of AP
+  adapters, domains or control domains. The objective of the handler is t
+  add the new queues accessible via the host in bulk rather than one queue
+  at a time via the driver's queue device probe callback. The locks and the
+  order in which they must be taken by this operation are:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held for the duration of the operation to:
+
+	* Iterate over the matrix_dev->mdev_list to find each matrix_mdev to
+	  which a queue device added is assigned and prevent other operations
+	  from modifying the list while processing each affected matrix_mdev
+	  object.
+
+	* Prevent other operations from acquiring access to the KVM pointer in
+	  each affected matrix_mdev instance (matrix_mdev->kvm) for the purpose
+	  of plugging/unplugging AP devices passed through to the KVM guest via
+	  that instance.
+
+  2. KVM Lock (kvm->lock):
+
+	This mutex must be held for each affected matrix_mdev instance while
+	the AP devices passed through to the KVM guest are plugged in to prevent
+	other operations from accessing the guest's state while it is in flux.
+
+	Note: This lock must be re-acquired for each matrix_mdev instance.
+
+  3. Matrix Devices Lock (matrix_dev->mdevs_lock):
+
+	This lock must be held for each affected matrix_mdev to prevent access
+	to the matrix_mdev state while writing/reading state values during the
+	operation, such as the masks used to construct the KVM guest's AP
+	configuration.
+
+	Note: This lock must be re-acquired for each matrix_mdev instance.
+
+Operation: Handle resource in use query:
+
+  The vfio_ap device driver registers a callback function with the AP bus to be
+  called when changes to the bus's sysfs /sys/bus/ap/apmask or
+  /sys/bus/ap/aqmask attributes would result in one or more AP queue devices
+  getting unbound from the vfio_ap device driver to verify none of them are in
+  use by the driver (i.e., assigned to a matrix_mdev instance). If this function
+  is called while an adapter or domain is also being assigned to a mediated
+  device, this could result in a deadlock; for example:
+
+  1. A system administrator assigns an adapter to a mediated device under the
+     control of the vfio_ap device driver. The driver will need to first take
+     the matrix_dev->guests_lock to potentially hot plug the adapter into
+     the KVM guest.
+  2. At the same time, a system administrator sets a bit in the sysfs
+     /sys/bus/ap/ap_mask attribute. To complete the operation, the AP bus
+     must:
+     a. Take the ap_perms_mutex lock to update the object storing the values
+        for the /sys/bus/ap/ap_mask attribute.
+     b. Call the vfio_ap device driver's in-use callback to verify that no
+        queues now being reserved for the default zcrypt drivers are
+        in use by the vfio_ap device driver. To do the verification, the in-use
+        callback function takes the matrix_dev->guests_lock, but has to wait
+        because it is already held by the operation in 1 above.
+  3. The vfio_ap device driver calls an AP bus function to verify that the
+     new queues resulting from the assignment of the adapter in step 1 are
+     not reserved for the default zcrypt device driver. This AP bus function
+     tries to take the ap_perms_mutex lock but gets stuck waiting for the
+     it due to step 2a above.
+
+    Consequently, we have the following deadlock situation:
+
+    matrix_dev->guests_lock locked (1)
+    ap_perms_mutex lock locked (2a)
+    Waiting for matrix_dev->lock (2b) which is currently held (1)
+    Waiting for ap_perms_mutex lock (3) which is currently held (2a)
+
+  To prevent the deadlock scenario, the in_use operation will take the
+  required locks using the mutex_trylock() function and if the lock can not be
+  acquired will terminate and return -EBUSY to indicate the driver is busy
+  processing another request.
+
+  The locks required to respond to the handle resource in use query and the
+  order in which they must be taken are:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+  This mutex must be held for the duration of the operation to iterate over the
+  matrix_dev->mdev_list to determine whether any of the queues to be unbound
+  are assigned to a matrix_mdev instance.
+
+  2. Matrix Devices Lock (matrix_dev->mdevs_lock):
+
+  This mutex must be held for the duration of the operation to ensure that the
+  AP configuration of each matrix_mdev instance does not change while verifying
+  that none of the queue devices to be removed from the vfio_ap driver are
+  assigned to it.
Signature · 1 line
2.31.1
author maintainer
#28 lore raw

PATCH v18 12/18 s390/vfio-ap: reset queues after adapter/domain unassignment

In reply to Tony Krowiak

When an adapter or domain is unassigned from an mdev providing the AP configuration to a running KVM guest, one or more of the guest's queues may get dynamically removed. Since the removed queues could get re-assigned to another mdev, they need to be reset. So, when an adapter or domain is unassigned from the mdev, the queues that are removed from the guest's AP configuration will be reset.

---
 drivers/s390/crypto/vfio_ap_ops.c     | 146 ++++++++++++++++++--------
 drivers/s390/crypto/vfio_ap_private.h |   2 +
 2 files changed, 104 insertions(+), 44 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +102 -44
index f0fdfbbe2d29..e9f7ec6fc6a5 100644
@@ -24,9 +24,10 @@
 #define VFIO_AP_MDEV_TYPE_HWVIRT "passthrough"
 #define VFIO_AP_MDEV_NAME_HWVIRT "VFIO AP Passthrough Device"
 
-static int vfio_ap_mdev_reset_queues(struct ap_matrix_mdev *matrix_mdev);
+static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable);
 static struct vfio_ap_queue *vfio_ap_find_queue(int apqn);
 static const struct vfio_device_ops vfio_ap_matrix_dev_ops;
+static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, unsigned int retry);
 
 /**
  * vfio_ap_mdev_get_queue - retrieve a queue with a specific APQN from a
@@ -356,6 +357,7 @@ static bool vfio_ap_mdev_filter_matrix(unsigned long *apm, unsigned long *aqm,
 	unsigned long apid, apqi, apqn;
 	DECLARE_BITMAP(shadow_apm, AP_DEVICES);
 	DECLARE_BITMAP(shadow_aqm, AP_DOMAINS);
+	struct vfio_ap_queue *q;
 
 	ret = ap_qci(&matrix_dev->info);
 	if (ret)
@@ -386,8 +388,8 @@ static bool vfio_ap_mdev_filter_matrix(unsigned long *apm, unsigned long *aqm,
 			 * hardware device.
 			 */
 			apqn = AP_MKQID(apid, apqi);
-
-			if (!vfio_ap_mdev_get_queue(matrix_mdev, apqn)) {
+			q = vfio_ap_mdev_get_queue(matrix_mdev, apqn);
+			if (!q || q->reset_rc) {
 				clear_bit_inv(apid,
 					      matrix_mdev->shadow_apcb.apm);
 				break;
@@ -471,12 +473,6 @@ static void vfio_ap_unlink_mdev_fr_queue(struct vfio_ap_queue *q)
 	q->matrix_mdev = NULL;
 }
 
-static void vfio_ap_mdev_unlink_queue(struct vfio_ap_queue *q)
-{
-	vfio_ap_unlink_queue_fr_mdev(q);
-	vfio_ap_unlink_mdev_fr_queue(q);
-}
-
 static void vfio_ap_mdev_unlink_fr_queues(struct ap_matrix_mdev *matrix_mdev)
 {
 	struct vfio_ap_queue *q;
@@ -501,7 +497,7 @@ static void vfio_ap_mdev_remove(struct mdev_device *mdev)
 
 	mutex_lock(&matrix_dev->guests_lock);
 	mutex_lock(&matrix_dev->mdevs_lock);
-	vfio_ap_mdev_reset_queues(matrix_mdev);
+	vfio_ap_mdev_reset_queues(&matrix_mdev->qtable);
 	vfio_ap_mdev_unlink_fr_queues(matrix_mdev);
 	list_del(&matrix_mdev->node);
 	mutex_unlock(&matrix_dev->mdevs_lock);
@@ -760,17 +756,61 @@ static ssize_t assign_adapter_store(struct device *dev,
 }
 static DEVICE_ATTR_WO(assign_adapter);
 
+static void vfio_ap_unlink_apqn_fr_mdev(struct ap_matrix_mdev *matrix_mdev,
+					unsigned long apid, unsigned long apqi,
+					struct ap_queue_table *qtable)
+{
+	struct vfio_ap_queue *q;
+
+	q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi));
+	/* If the queue is assigned to the matrix mdev, unlink it. */
+	if (q)
+		vfio_ap_unlink_queue_fr_mdev(q);
+
+	/* If the queue is assigned to the APCB, store it in @qtable. */
+	if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) &&
+	    test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm))
+		hash_add(qtable->queues, &q->mdev_qnode, q->apqn);
+}
+
+/**
+ * vfio_ap_mdev_unlink_adapter - unlink all queues associated with unassigned
+ *				 adapter from the matrix mdev to which the
+ *				 adapter was assigned.
+ * @matrix_mdev: the matrix mediated device to which the adapter was assigned.
+ * @apid: the APID of the unassigned adapter.
+ * @qtable: table for storing queues associated with unassigned adapter.
+ */
 static void vfio_ap_mdev_unlink_adapter(struct ap_matrix_mdev *matrix_mdev,
-					unsigned long apid)
+					unsigned long apid,
+					struct ap_queue_table *qtable)
 {
 	unsigned long apqi;
+
+	for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS)
+		vfio_ap_unlink_apqn_fr_mdev(matrix_mdev, apid, apqi, qtable);
+}
+
+static void vfio_ap_mdev_hot_unplug_adapter(struct ap_matrix_mdev *matrix_mdev,
+					    unsigned long apid)
+{
+	int bkt;
 	struct vfio_ap_queue *q;
+	struct ap_queue_table qtable;
+
+	hash_init(qtable.queues);
+	vfio_ap_mdev_unlink_adapter(matrix_mdev, apid, &qtable);
+
+	if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm)) {
+		clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm);
+		vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+	}
 
-	for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) {
-		q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi));
+	vfio_ap_mdev_reset_queues(&qtable);
 
-		if (q)
-			vfio_ap_mdev_unlink_queue(q);
+	hash_for_each(qtable.queues, bkt, q, mdev_qnode) {
+		vfio_ap_unlink_mdev_fr_queue(q);
+		hash_del(&q->mdev_qnode);
 	}
 }
 
@@ -809,13 +849,7 @@ static ssize_t unassign_adapter_store(struct device *dev,
 	}
 
 	clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm);
-	vfio_ap_mdev_unlink_adapter(matrix_mdev, apid);
-
-	if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm)) {
-		clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm);
-		vfio_ap_mdev_hotplug_apcb(matrix_mdev);
-	}
-
+	vfio_ap_mdev_hot_unplug_adapter(matrix_mdev, apid);
 	ret = count;
 done:
 	vfio_ap_mdev_put_locks(matrix_mdev);
@@ -909,16 +943,35 @@ static ssize_t assign_domain_store(struct device *dev,
 static DEVICE_ATTR_WO(assign_domain);
 
 static void vfio_ap_mdev_unlink_domain(struct ap_matrix_mdev *matrix_mdev,
-				       unsigned long apqi)
+				       unsigned long apqi,
+				       struct ap_queue_table *qtable)
 {
 	unsigned long apid;
+
+	for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES)
+		vfio_ap_unlink_apqn_fr_mdev(matrix_mdev, apid, apqi, qtable);
+}
+
+static void vfio_ap_mdev_hot_unplug_domain(struct ap_matrix_mdev *matrix_mdev,
+					   unsigned long apqi)
+{
+	int bkt;
 	struct vfio_ap_queue *q;
+	struct ap_queue_table qtable;
 
-	for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, AP_DEVICES) {
-		q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi));
+	hash_init(qtable.queues);
+	vfio_ap_mdev_unlink_domain(matrix_mdev, apqi, &qtable);
 
-		if (q)
-			vfio_ap_mdev_unlink_queue(q);
+	if (test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) {
+		clear_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm);
+		vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+	}
+
+	vfio_ap_mdev_reset_queues(&qtable);
+
+	hash_for_each(qtable.queues, bkt, q, mdev_qnode) {
+		vfio_ap_unlink_mdev_fr_queue(q);
+		hash_del(&q->mdev_qnode);
 	}
 }
 
@@ -957,13 +1010,7 @@ static ssize_t unassign_domain_store(struct device *dev,
 	}
 
 	clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm);
-	vfio_ap_mdev_unlink_domain(matrix_mdev, apqi);
-
-	if (test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) {
-		clear_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm);
-		vfio_ap_mdev_hotplug_apcb(matrix_mdev);
-	}
-
+	vfio_ap_mdev_hot_unplug_domain(matrix_mdev, apqi);
 	ret = count;
 done:
 	vfio_ap_mdev_put_locks(matrix_mdev);
@@ -1273,9 +1320,9 @@ static void vfio_ap_mdev_unset_kvm(struct ap_matrix_mdev *matrix_mdev,
 		mutex_lock(&kvm->lock);
 		mutex_lock(&matrix_dev->mdevs_lock);
 
-		kvm_arch_crypto_clear_masks(kvm);
-		vfio_ap_mdev_reset_queues(matrix_mdev);
-		kvm_put_kvm(kvm);
+		kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
+		vfio_ap_mdev_reset_queues(&matrix_mdev->qtable);
+		kvm_put_kvm(matrix_mdev->kvm);
 		matrix_mdev->kvm = NULL;
 
 		mutex_unlock(&matrix_dev->mdevs_lock);
@@ -1328,14 +1375,17 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, unsigned int retry)
 
 	if (!q)
 		return 0;
+	q->reset_rc = 0;
 
 retry_zapq:
 	status = ap_zapq(q->apqn);
 	switch (status.response_code) {
 	case AP_RESPONSE_NORMAL:
 		ret = 0;
+		q->reset_rc = status.response_code;
 		break;
 	case AP_RESPONSE_RESET_IN_PROGRESS:
+		q->reset_rc = status.response_code;
 		if (retry--) {
 			msleep(20);
 			goto retry_zapq;
@@ -1345,13 +1395,20 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, unsigned int retry)
 	case AP_RESPONSE_Q_NOT_AVAIL:
 	case AP_RESPONSE_DECONFIGURED:
 	case AP_RESPONSE_CHECKSTOPPED:
-		WARN_ON_ONCE(status.irq_enabled);
+		WARN_ONCE(status.irq_enabled,
+			  "PQAP/ZAPQ for %02x.%04x failed with rc=%u while IRQ enabled",
+			  AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
+			  status.response_code);
+		q->reset_rc = status.response_code;
 		ret = -EBUSY;
 		goto free_resources;
 	default:
 		/* things are really broken, give up */
-		WARN(true, "PQAP/ZAPQ completed with invalid rc (%x)\n",
+		WARN(true,
+		     "PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n",
+		     AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
 		     status.response_code);
+		q->reset_rc = status.response_code;
 		return -EIO;
 	}
 
@@ -1362,7 +1419,8 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, unsigned int retry)
 		msleep(20);
 		status = ap_tapq(q->apqn, NULL);
 	}
-	WARN_ON_ONCE(retry2 <= 0);
+	WARN_ONCE(retry2 <= 0, "unable to verify reset of queue %02x.%04x",
+		  AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn));
 
 free_resources:
 	vfio_ap_free_aqic_resources(q);
@@ -1370,12 +1428,12 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, unsigned int retry)
 	return ret;
 }
 
-static int vfio_ap_mdev_reset_queues(struct ap_matrix_mdev *matrix_mdev)
+static int vfio_ap_mdev_reset_queues(struct ap_queue_table *qtable)
 {
-	int ret, bkt, rc = 0;
+	int rc = 0, ret, bkt;
 	struct vfio_ap_queue *q;
 
-	hash_for_each(matrix_mdev->qtable.queues, bkt, q, mdev_qnode) {
+	hash_for_each(qtable->queues, bkt, q, mdev_qnode) {
 		ret = vfio_ap_mdev_reset_queue(q, 1);
 		/*
 		 * Regardless whether a queue turns out to be busy, or
@@ -1463,7 +1521,7 @@ static ssize_t vfio_ap_mdev_ioctl(struct vfio_device *vdev,
 		ret = vfio_ap_mdev_get_device_info(arg);
 		break;
 	case VFIO_DEVICE_RESET:
-		ret = vfio_ap_mdev_reset_queues(matrix_mdev);
+		ret = vfio_ap_mdev_reset_queues(&matrix_mdev->qtable);
 		break;
 	default:
 		ret = -EOPNOTSUPP;
drivers/s390/crypto/vfio_ap_private.h +2 -0
index 7e82a72d2083..a3811cd9852d 100644
@@ -128,6 +128,7 @@ struct ap_matrix_mdev {
  * @apqn: the APQN of the AP queue device
  * @saved_isc: the guest ISC registered with the GIB interface
  * @mdev_qnode: allows the vfio_ap_queue struct to be added to a hashtable
+ * @reset_rc: the status response code from the last reset of the queue
  */
 struct vfio_ap_queue {
 	struct ap_matrix_mdev *matrix_mdev;
@@ -136,6 +137,7 @@ struct vfio_ap_queue {
 #define VFIO_AP_ISC_INVALID 0xff
 	unsigned char saved_isc;
 	struct hlist_node mdev_qnode;
+	unsigned int reset_rc;
 };
 
 int vfio_ap_mdev_register(void);
Signature · 1 line
2.31.1
author maintainer
#38 lore raw

PATCH v18 15/18 s390/vfio-ap: handle config changed and scan complete notification

In reply to Tony Krowiak

This patch implements two new AP driver callbacks:

void (*on_config_changed)(struct ap_config_info *new_config_info,

                      struct ap_config_info *old_config_info);

void (*on_scan_complete)(struct ap_config_info *new_config_info,

                     struct ap_config_info *old_config_info);

The on_config_changed callback is invoked at the start of the AP bus scan function when it determines that the host AP configuration information has changed since the previous scan.

The vfio_ap device driver registers a callback function for this callback that performs the following operations:

1. Unplugs the adapters, domains and control domains removed from the host's AP configuration from the guests to which they are assigned in a single operation.

2. Stores bitmaps identifying the adapters, domains and control domains added to the host's AP configuration with the structure representing the mediated device. When the vfio_ap device driver's probe callback is subsequently invoked, the probe function will recognize that the queue is being probed due to a change in the host's AP configuration and the plugging of the queue into the guest will be bypassed.

The on_scan_complete callback is invoked after the ap bus scan is completed if the host AP configuration data has changed. The vfio_ap device driver registers a callback function for this callback that hot plugs each queue and control domain added to the AP configuration for each guest using them in a single hot plug operation.

---
 drivers/s390/crypto/vfio_ap_drv.c     |   4 +-
 drivers/s390/crypto/vfio_ap_ops.c     | 347 ++++++++++++++++++++++++--
 drivers/s390/crypto/vfio_ap_private.h |  15 +-
 3 files changed, 338 insertions(+), 28 deletions(-)
drivers/s390/crypto/vfio_ap_drv.c +3 -1
index c4f0e28b1e91..711d5f2645b8 100644
@@ -104,6 +104,8 @@ static struct ap_driver vfio_ap_drv = {
 	.probe = vfio_ap_mdev_probe_queue,
 	.remove = vfio_ap_mdev_remove_queue,
 	.in_use = vfio_ap_mdev_resource_in_use,
+	.on_config_changed = vfio_ap_on_cfg_changed,
+	.on_scan_complete = vfio_ap_on_scan_complete,
 	.ids = ap_queue_ids,
 };
 
@@ -151,7 +153,7 @@ static int vfio_ap_matrix_dev_create(void)
 
 	/* Fill in config info via PQAP(QCI), if available */
 	if (test_facility(12)) {
-		ret = ap_qci(&matrix_dev->info);
+		ret = ap_qci(&matrix_dev->config_info);
 		if (ret)
 			goto matrix_alloc_err;
 	}
drivers/s390/crypto/vfio_ap_ops.c +322 -25

428 lines not on this page · raw message

drivers/s390/crypto/vfio_ap_private.h +13 -2
index e30f171f89ec..a7fe4a3cf896 100644
@@ -31,7 +31,9 @@
  *
  * @device:	generic device structure associated with the AP matrix device
  * @available_instances: number of mediated matrix devices that can be created
- * @info:	the struct containing the output from the PQAP(QCI) instruction
+ * @config_info: the object containing the current AP configuration for the host
+ * @config_info_prev: the object containing the previous AP configuration for
+ *		      Gthe host
  * @mdev_list:	the list of mediated matrix devices created
  * @mdevs_lock:	mutex for locking the ap_matrix_mdev devices under the control
  *		of the vfio_ap device driver. This lock will be taken every time
@@ -47,7 +49,8 @@
 struct ap_matrix_dev {
 	struct device device;
 	atomic_t available_instances;
-	struct ap_config_info info;
+	struct ap_config_info config_info;
+	struct ap_config_info config_info_prev;
 	struct list_head mdev_list;
 	struct mutex mdevs_lock;
 	struct ap_driver  *vfio_ap_drv;
@@ -118,6 +121,9 @@ struct ap_matrix_mdev {
 	crypto_hook pqap_hook;
 	struct mdev_device *mdev;
 	struct ap_queue_table qtable;
+	DECLARE_BITMAP(apm_add, AP_DEVICES);
+	DECLARE_BITMAP(aqm_add, AP_DOMAINS);
+	DECLARE_BITMAP(adm_add, AP_DOMAINS);
 };
 
 /**
@@ -148,4 +154,9 @@ void vfio_ap_mdev_remove_queue(struct ap_device *queue);
 
 int vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm);
 
+void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info,
+			    struct ap_config_info *old_config_info);
+void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info,
+			      struct ap_config_info *old_config_info);
+
 #endif /* _VFIO_AP_PRIVATE_H_ */
Signature · 1 line
2.31.1
author maintainer
#7 lore raw

PATCH v18 7/18 s390/vfio-ap: refresh guest's APCB by filtering APQNs assigned to mdev

In reply to Tony Krowiak

Refresh the guest's APCB by filtering the APQNs assigned to the matrix mdev that do not reference an AP queue device bound to the vfio_ap device driver. The mdev's APQNs will be filtered according to the following rules:

* The APID of each adapter and the APQI of each domain that is not in the host's AP configuration is filtered out.

* The APID of each adapter comprising an APQN that does not reference a queue device bound to the vfio_ap device driver is filtered. The APQNs are derived from the Cartesian product of the APID of each adapter and APQI of each domain assigned to the mdev.

The control domains that are not assigned to the host's AP configuration will also be filtered before assigning them to the guest's APCB.

---
 drivers/s390/crypto/vfio_ap_ops.c | 96 ++++++++++++++++++++++++++++++-
 1 file changed, 93 insertions(+), 3 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +93 -3
index 4b676a55f203..b67b2f0faeea 100644
@@ -317,6 +317,63 @@ static void vfio_ap_matrix_init(struct ap_config_info *info,
 	matrix->adm_max = info->apxa ? info->Nd : 15;
 }
 
+static void vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev *matrix_mdev)
+{
+	bitmap_and(matrix_mdev->shadow_apcb.adm, matrix_mdev->matrix.adm,
+		   (unsigned long *)matrix_dev->info.adm, AP_DOMAINS);
+}
+
+/*
+ * vfio_ap_mdev_filter_matrix - copy the mdev's AP configuration to the KVM
+ *				guest's APCB then filter the APIDs that do not
+ *				comprise at least one APQN that references a
+ *				queue device bound to the vfio_ap device driver.
+ *
+ * @matrix_mdev: the mdev whose AP configuration is to be filtered.
+ */
+static void vfio_ap_mdev_filter_matrix(unsigned long *apm, unsigned long *aqm,
+				       struct ap_matrix_mdev *matrix_mdev)
+{
+	int ret;
+	unsigned long apid, apqi, apqn;
+
+	ret = ap_qci(&matrix_dev->info);
+	if (ret)
+		return;
+
+	vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_apcb);
+
+	/*
+	 * Copy the adapters, domains and control domains to the shadow_apcb
+	 * from the matrix mdev, but only those that are assigned to the host's
+	 * AP configuration.
+	 */
+	bitmap_and(matrix_mdev->shadow_apcb.apm, matrix_mdev->matrix.apm,
+		   (unsigned long *)matrix_dev->info.apm, AP_DEVICES);
+	bitmap_and(matrix_mdev->shadow_apcb.aqm, matrix_mdev->matrix.aqm,
+		   (unsigned long *)matrix_dev->info.aqm, AP_DOMAINS);
+
+	for_each_set_bit_inv(apid, apm, AP_DEVICES) {
+		for_each_set_bit_inv(apqi, aqm, AP_DOMAINS) {
+			/*
+			 * If the APQN is not bound to the vfio_ap device
+			 * driver, then we can't assign it to the guest's
+			 * AP configuration. The AP architecture won't
+			 * allow filtering of a single APQN, so let's filter
+			 * the APID since an adapter represents a physical
+			 * hardware device.
+			 */
+			apqn = AP_MKQID(apid, apqi);
+
+			if (!vfio_ap_mdev_get_queue(matrix_mdev, apqn)) {
+				clear_bit_inv(apid,
+					      matrix_mdev->shadow_apcb.apm);
+				break;
+			}
+		}
+	}
+}
+
 static int vfio_ap_mdev_probe(struct mdev_device *mdev)
 {
 	struct ap_matrix_mdev *matrix_mdev;
@@ -672,6 +729,8 @@ static ssize_t assign_adapter_store(struct device *dev,
 {
 	int ret;
 	unsigned long apid;
+	DECLARE_BITMAP(apm, AP_DEVICES);
+
 	struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev);
 
 	mutex_lock(&matrix_dev->lock);
@@ -701,12 +760,15 @@ static ssize_t assign_adapter_store(struct device *dev,
 		goto done;
 
 	set_bit_inv(apid, matrix_mdev->matrix.apm);
+	memset(apm, 0, sizeof(apm));
+	set_bit_inv(apid, apm);
 
 	ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev);
 	if (ret)
 		goto share_err;
 
 	vfio_ap_mdev_link_adapter(matrix_mdev, apid);
+	vfio_ap_mdev_filter_matrix(apm, matrix_mdev->matrix.aqm, matrix_mdev);
 	ret = count;
 	goto done;
 
@@ -775,6 +837,10 @@ static ssize_t unassign_adapter_store(struct device *dev,
 
 	clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm);
 	vfio_ap_mdev_unlink_adapter(matrix_mdev, apid);
+
+	if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm))
+		clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm);
+
 	ret = count;
 done:
 	mutex_unlock(&matrix_dev->lock);
@@ -848,6 +914,7 @@ static ssize_t assign_domain_store(struct device *dev,
 {
 	int ret;
 	unsigned long apqi;
+	DECLARE_BITMAP(aqm, AP_DOMAINS);
 	struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev);
 	unsigned long max_apqi = matrix_mdev->matrix.aqm_max;
 
@@ -872,12 +939,15 @@ static ssize_t assign_domain_store(struct device *dev,
 		goto done;
 
 	set_bit_inv(apqi, matrix_mdev->matrix.aqm);
+	memset(aqm, 0, sizeof(aqm));
+	set_bit_inv(apqi, aqm);
 
 	ret = vfio_ap_mdev_verify_no_sharing(matrix_mdev);
 	if (ret)
 		goto share_err;
 
 	vfio_ap_mdev_link_domain(matrix_mdev, apqi);
+	vfio_ap_mdev_filter_matrix(matrix_mdev->matrix.apm, aqm, matrix_mdev);
 	ret = count;
 	goto done;
 
@@ -946,6 +1016,10 @@ static ssize_t unassign_domain_store(struct device *dev,
 
 	clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm);
 	vfio_ap_mdev_unlink_domain(matrix_mdev, apqi);
+
+	if (test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm))
+		clear_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm);
+
 	ret = count;
 
 done:
@@ -999,6 +1073,7 @@ static ssize_t assign_control_domain_store(struct device *dev,
 	 * number of control domains that can be assigned.
 	 */
 	set_bit_inv(id, matrix_mdev->matrix.adm);
+	vfio_ap_mdev_filter_cdoms(matrix_mdev);
 	ret = count;
 done:
 	mutex_unlock(&matrix_dev->lock);
@@ -1046,6 +1121,10 @@ static ssize_t unassign_control_domain_store(struct device *dev,
 	}
 
 	clear_bit_inv(domid, matrix_mdev->matrix.adm);
+
+	if (test_bit_inv(domid, matrix_mdev->shadow_apcb.adm))
+		clear_bit_inv(domid, matrix_mdev->shadow_apcb.adm);
+
 	ret = count;
 done:
 	mutex_unlock(&matrix_dev->lock);
@@ -1186,8 +1265,6 @@ static int vfio_ap_mdev_set_kvm(struct ap_matrix_mdev *matrix_mdev,
 
 		kvm_get_kvm(kvm);
 		matrix_mdev->kvm = kvm;
-		memcpy(&matrix_mdev->shadow_apcb, &matrix_mdev->matrix,
-		       sizeof(struct ap_matrix));
 		kvm_arch_crypto_set_masks(kvm, matrix_mdev->shadow_apcb.apm,
 					  matrix_mdev->shadow_apcb.aqm,
 					  matrix_mdev->shadow_apcb.adm);
@@ -1528,6 +1605,7 @@ static void vfio_ap_queue_link_mdev(struct vfio_ap_queue *q)
 int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
 {
 	struct vfio_ap_queue *q;
+	DECLARE_BITMAP(apm, AP_DEVICES);
 
 	q = kzalloc(sizeof(*q), GFP_KERNEL);
 	if (!q)
@@ -1537,6 +1615,12 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
 	q->apqn = to_ap_queue(&apdev->device)->qid;
 	q->saved_isc = VFIO_AP_ISC_INVALID;
 	vfio_ap_queue_link_mdev(q);
+	if (q->matrix_mdev) {
+		memset(apm, 0, sizeof(apm));
+		set_bit_inv(AP_QID_CARD(q->apqn), apm);
+		vfio_ap_mdev_filter_matrix(apm, q->matrix_mdev->matrix.aqm,
+					   q->matrix_mdev);
+	}
 	dev_set_drvdata(&apdev->device, q);
 	mutex_unlock(&matrix_dev->lock);
 
@@ -1545,14 +1629,20 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
 
 void vfio_ap_mdev_remove_queue(struct ap_device *apdev)
 {
+	unsigned long apid;
 	struct vfio_ap_queue *q;
 
 	mutex_lock(&matrix_dev->lock);
 	q = dev_get_drvdata(&apdev->device);
 
-	if (q->matrix_mdev)
+	if (q->matrix_mdev) {
 		vfio_ap_unlink_queue_fr_mdev(q);
 
+		apid = AP_QID_CARD(q->apqn);
+		if (test_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm))
+			clear_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm);
+	}
+
 	vfio_ap_mdev_reset_queue(q, 1);
 	dev_set_drvdata(&apdev->device, NULL);
 	kfree(q);
Signature · 1 line
2.31.1
author maintainer
#22 lore raw

PATCH v18 10/18 s390/vfio-ap: allow hot plug/unplug of AP devices when assigned/unassigned

In reply to Tony Krowiak

Let's allow adapters, domains and control domains to be hot plugged into and hot unplugged from a KVM guest using a matrix mdev when an adapter, domain or control domain is assigned to or unassigned from the matrix mdev.

Whenever an assignment or unassignment of an adapter, domain or control domain is performed, the AP configuration assigned to the matrix mediated device will be filtered and assigned to the AP control block (APCB) that supplies the AP configuration to the guest so that no adapter, domain or control domain that is not in the host's AP configuration nor any APQN that does not reference a queue device bound to the vfio_ap device driver is assigned.

After updating the APCB, if the mdev is in use by a KVM guest, it is hot plugged into the guest to dynamically provide access to the adapters, domains and control domains provided via the newly refreshed APCB.

Keep in mind that the matrix_dev->guests_lock must be taken outside of the matrix_mdev->kvm->lock which in turn must be taken outside of the matrix_dev->mdevs_lock in order to avoid circular lock dependencies (i.e., a lockdep splat).Consequently, the locking order for hot plugging the guest's APCB must be:

matrix_dev->guests_lock => matrix_mdev->kvm->lock => matrix_dev->mdevs_lock

---
 drivers/s390/crypto/vfio_ap_ops.c | 198 +++++++++++++++++++-----------
 1 file changed, 125 insertions(+), 73 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +125 -73
index 623a4b38676d..4c382cd3afc7 100644
@@ -317,10 +317,25 @@ static void vfio_ap_matrix_init(struct ap_config_info *info,
 	matrix->adm_max = info->apxa ? info->Nd : 15;
 }
 
-static void vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev *matrix_mdev)
+static void vfio_ap_mdev_hotplug_apcb(struct ap_matrix_mdev *matrix_mdev)
 {
+	if (matrix_mdev->kvm)
+		kvm_arch_crypto_set_masks(matrix_mdev->kvm,
+					  matrix_mdev->shadow_apcb.apm,
+					  matrix_mdev->shadow_apcb.aqm,
+					  matrix_mdev->shadow_apcb.adm);
+}
+
+static bool vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev *matrix_mdev)
+{
+	DECLARE_BITMAP(shadow_adm, AP_DOMAINS);
+
+	bitmap_copy(shadow_adm, matrix_mdev->shadow_apcb.adm, AP_DOMAINS);
 	bitmap_and(matrix_mdev->shadow_apcb.adm, matrix_mdev->matrix.adm,
 		   (unsigned long *)matrix_dev->info.adm, AP_DOMAINS);
+
+	return !bitmap_equal(shadow_adm, matrix_mdev->shadow_apcb.adm,
+			     AP_DOMAINS);
 }
 
 /*
@@ -330,17 +345,24 @@ static void vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev *matrix_mdev)
  *				queue device bound to the vfio_ap device driver.
  *
  * @matrix_mdev: the mdev whose AP configuration is to be filtered.
+ *
+ * Return: a boolean value indicating whether the KVM guest's APCB was changed
+ *	   by the filtering or not.
  */
-static void vfio_ap_mdev_filter_matrix(unsigned long *apm, unsigned long *aqm,
+static bool vfio_ap_mdev_filter_matrix(unsigned long *apm, unsigned long *aqm,
 				       struct ap_matrix_mdev *matrix_mdev)
 {
 	int ret;
 	unsigned long apid, apqi, apqn;
+	DECLARE_BITMAP(shadow_apm, AP_DEVICES);
+	DECLARE_BITMAP(shadow_aqm, AP_DOMAINS);
 
 	ret = ap_qci(&matrix_dev->info);
 	if (ret)
-		return;
+		return false;
 
+	bitmap_copy(shadow_apm, matrix_mdev->shadow_apcb.apm, AP_DEVICES);
+	bitmap_copy(shadow_aqm, matrix_mdev->shadow_apcb.aqm, AP_DOMAINS);
 	vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_apcb);
 
 	/*
@@ -372,6 +394,11 @@ static void vfio_ap_mdev_filter_matrix(unsigned long *apm, unsigned long *aqm,
 			}
 		}
 	}
+
+	return !bitmap_equal(shadow_apm, matrix_mdev->shadow_apcb.apm,
+			     AP_DEVICES) ||
+	       !bitmap_equal(shadow_aqm, matrix_mdev->shadow_apcb.aqm,
+			     AP_DOMAINS);
 }
 
 static int vfio_ap_mdev_probe(struct mdev_device *mdev)
@@ -472,11 +499,13 @@ static void vfio_ap_mdev_remove(struct mdev_device *mdev)
 
 	vfio_unregister_group_dev(&matrix_mdev->vdev);
 
+	mutex_lock(&matrix_dev->guests_lock);
 	mutex_lock(&matrix_dev->mdevs_lock);
 	vfio_ap_mdev_reset_queues(matrix_mdev);
 	vfio_ap_mdev_unlink_fr_queues(matrix_mdev);
 	list_del(&matrix_mdev->node);
 	mutex_unlock(&matrix_dev->mdevs_lock);
+	mutex_unlock(&matrix_dev->guests_lock);
 	vfio_uninit_group_dev(&matrix_mdev->vdev);
 	kfree(matrix_mdev);
 	atomic_inc(&matrix_dev->available_instances);
@@ -612,6 +641,51 @@ static void vfio_ap_mdev_link_adapter(struct ap_matrix_mdev *matrix_mdev,
 				       AP_MKQID(apid, apqi));
 }
 
+/**
+ * vfio_ap_mdev_get_locks - acquire the locks required to assign/unassign AP
+ *			    adapters, domains and control domains for an mdev in
+ *			    the proper locking order.
+ *
+ * @matrix_mdev: the matrix mediated device object
+ */
+static void vfio_ap_mdev_get_locks(struct ap_matrix_mdev *matrix_mdev)
+{
+	/* Lock the mutex required to access the KVM guest's state */
+	mutex_lock(&matrix_dev->guests_lock);
+
+	/* If a KVM guest is running, lock the mutex required to plug/unplug the
+	 * AP devices passed through to the guest
+	 */
+	if (matrix_mdev->kvm)
+		mutex_lock(&matrix_mdev->kvm->lock);
+
+	/* The lock required to access the mdev's state */
+	mutex_lock(&matrix_dev->mdevs_lock);
+}
+
+/**
+ * vfio_ap_mdev_put_locks - release the locks used to assign/unassign AP
+ *			    adapters, domains and control domains in the proper
+ *			    unlocking order.
+ *
+ * @matrix_mdev: the matrix mediated device object
+ */
+static void vfio_ap_mdev_put_locks(struct ap_matrix_mdev *matrix_mdev)
+{
+	/* Unlock the mutex taken to access the matrix_mdev's state */
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	/*
+	 * If a KVM guest is running, unlock the mutex taken to plug/unplug the
+	 * AP devices passed through to the guest.
+	 */
+	if (matrix_mdev->kvm)
+		mutex_unlock(&matrix_mdev->kvm->lock);
+
+	/* Unlock the mutex taken to allow access to the KVM guest's state */
+	mutex_unlock(&matrix_dev->guests_lock);
+}
+
 /**
  * assign_adapter_store - parses the APID from @buf and sets the
  * corresponding bit in the mediated matrix device's APM
@@ -649,17 +723,9 @@ static ssize_t assign_adapter_store(struct device *dev,
 	int ret;
 	unsigned long apid;
 	DECLARE_BITMAP(apm, AP_DEVICES);
-
 	struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev);
 
-	mutex_lock(&matrix_dev->guests_lock);
-	mutex_lock(&matrix_dev->mdevs_lock);
-
-	/* If the KVM guest is running, disallow assignment of adapter */
-	if (matrix_mdev->kvm) {
-		ret = -EBUSY;
-		goto done;
-	}
+	vfio_ap_mdev_get_locks(matrix_mdev);
 
 	ret = kstrtoul(buf, 0, &apid);
 	if (ret)
@@ -671,8 +737,6 @@ static ssize_t assign_adapter_store(struct device *dev,
 	}
 
 	set_bit_inv(apid, matrix_mdev->matrix.apm);
-	memset(apm, 0, sizeof(apm));
-	set_bit_inv(apid, apm);
 
 	ret = vfio_ap_mdev_validate_masks(matrix_mdev);
 	if (ret) {
@@ -680,12 +744,17 @@ static ssize_t assign_adapter_store(struct device *dev,
 		goto done;
 	}
 
+	memset(apm, 0, sizeof(apm));
+	set_bit_inv(apid, apm);
 	vfio_ap_mdev_link_adapter(matrix_mdev, apid);
-	vfio_ap_mdev_filter_matrix(apm, matrix_mdev->matrix.aqm, matrix_mdev);
+
+	if (vfio_ap_mdev_filter_matrix(apm,
+				       matrix_mdev->matrix.aqm, matrix_mdev))
+		vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+
 	ret = count;
 done:
-	mutex_unlock(&matrix_dev->mdevs_lock);
-	mutex_unlock(&matrix_dev->guests_lock);
+	vfio_ap_mdev_put_locks(matrix_mdev);
 
 	return ret;
 }
@@ -728,13 +797,7 @@ static ssize_t unassign_adapter_store(struct device *dev,
 	unsigned long apid;
 	struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev);
 
-	mutex_lock(&matrix_dev->mdevs_lock);
-
-	/* If the KVM guest is running, disallow unassignment of adapter */
-	if (matrix_mdev->kvm) {
-		ret = -EBUSY;
-		goto done;
-	}
+	vfio_ap_mdev_get_locks(matrix_mdev);
 
 	ret = kstrtoul(buf, 0, &apid);
 	if (ret)
@@ -748,12 +811,15 @@ static ssize_t unassign_adapter_store(struct device *dev,
 	clear_bit_inv((unsigned long)apid, matrix_mdev->matrix.apm);
 	vfio_ap_mdev_unlink_adapter(matrix_mdev, apid);
 
-	if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm))
+	if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm)) {
 		clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm);
+		vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+	}
 
 	ret = count;
 done:
-	mutex_unlock(&matrix_dev->mdevs_lock);
+	vfio_ap_mdev_put_locks(matrix_mdev);
+
 	return ret;
 }
 static DEVICE_ATTR_WO(unassign_adapter);
@@ -806,28 +872,19 @@ static ssize_t assign_domain_store(struct device *dev,
 	unsigned long apqi;
 	DECLARE_BITMAP(aqm, AP_DOMAINS);
 	struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev);
-	unsigned long max_apqi = matrix_mdev->matrix.aqm_max;
 
-	mutex_lock(&matrix_dev->guests_lock);
-	mutex_lock(&matrix_dev->mdevs_lock);
-
-	/* If the KVM guest is running, disallow assignment of domain */
-	if (matrix_mdev->kvm) {
-		ret = -EBUSY;
-		goto done;
-	}
+	vfio_ap_mdev_get_locks(matrix_mdev);
 
 	ret = kstrtoul(buf, 0, &apqi);
 	if (ret)
 		goto done;
-	if (apqi > max_apqi) {
+
+	if (apqi > matrix_mdev->matrix.apm_max) {
 		ret = -ENODEV;
 		goto done;
 	}
 
 	set_bit_inv(apqi, matrix_mdev->matrix.aqm);
-	memset(aqm, 0, sizeof(aqm));
-	set_bit_inv(apqi, aqm);
 
 	ret = vfio_ap_mdev_validate_masks(matrix_mdev);
 	if (ret) {
@@ -835,12 +892,17 @@ static ssize_t assign_domain_store(struct device *dev,
 		goto done;
 	}
 
+	memset(aqm, 0, sizeof(aqm));
+	set_bit_inv(apqi, aqm);
 	vfio_ap_mdev_link_domain(matrix_mdev, apqi);
-	vfio_ap_mdev_filter_matrix(matrix_mdev->matrix.apm, aqm, matrix_mdev);
+
+	if (vfio_ap_mdev_filter_matrix(matrix_mdev->matrix.apm, aqm,
+				       matrix_mdev))
+		vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+
 	ret = count;
 done:
-	mutex_unlock(&matrix_dev->mdevs_lock);
-	mutex_unlock(&matrix_dev->guests_lock);
+	vfio_ap_mdev_put_locks(matrix_mdev);
 
 	return ret;
 }
@@ -883,19 +945,13 @@ static ssize_t unassign_domain_store(struct device *dev,
 	unsigned long apqi;
 	struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev);
 
-	mutex_lock(&matrix_dev->mdevs_lock);
-
-	/* If the KVM guest is running, disallow unassignment of domain */
-	if (matrix_mdev->kvm) {
-		ret = -EBUSY;
-		goto done;
-	}
+	vfio_ap_mdev_get_locks(matrix_mdev);
 
 	ret = kstrtoul(buf, 0, &apqi);
 	if (ret)
 		goto done;
 
-	if (apqi > matrix_mdev->matrix.aqm_max) {
+	if (apqi > matrix_mdev->matrix.apm_max) {
 		ret = -ENODEV;
 		goto done;
 	}
@@ -903,13 +959,15 @@ static ssize_t unassign_domain_store(struct device *dev,
 	clear_bit_inv((unsigned long)apqi, matrix_mdev->matrix.aqm);
 	vfio_ap_mdev_unlink_domain(matrix_mdev, apqi);
 
-	if (test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm))
+	if (test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) {
 		clear_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm);
+		vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+	}
 
 	ret = count;
-
 done:
-	mutex_unlock(&matrix_dev->mdevs_lock);
+	vfio_ap_mdev_put_locks(matrix_mdev);
+
 	return ret;
 }
 static DEVICE_ATTR_WO(unassign_domain);
@@ -936,19 +994,13 @@ static ssize_t assign_control_domain_store(struct device *dev,
 	unsigned long id;
 	struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev);
 
-	mutex_lock(&matrix_dev->mdevs_lock);
-
-	/* If the KVM guest is running, disallow assignment of control domain */
-	if (matrix_mdev->kvm) {
-		ret = -EBUSY;
-		goto done;
-	}
+	vfio_ap_mdev_get_locks(matrix_mdev);
 
 	ret = kstrtoul(buf, 0, &id);
 	if (ret)
 		goto done;
 
-	if (id > matrix_mdev->matrix.adm_max) {
+	if (id > matrix_mdev->matrix.apm_max) {
 		ret = -ENODEV;
 		goto done;
 	}
@@ -959,10 +1011,13 @@ static ssize_t assign_control_domain_store(struct device *dev,
 	 * number of control domains that can be assigned.
 	 */
 	set_bit_inv(id, matrix_mdev->matrix.adm);
-	vfio_ap_mdev_filter_cdoms(matrix_mdev);
+	if (vfio_ap_mdev_filter_cdoms(matrix_mdev))
+		vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+
 	ret = count;
 done:
-	mutex_unlock(&matrix_dev->mdevs_lock);
+	vfio_ap_mdev_put_locks(matrix_mdev);
+
 	return ret;
 }
 static DEVICE_ATTR_WO(assign_control_domain);
@@ -988,32 +1043,29 @@ static ssize_t unassign_control_domain_store(struct device *dev,
 	int ret;
 	unsigned long domid;
 	struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev);
-	unsigned long max_domid =  matrix_mdev->matrix.adm_max;
 
-	mutex_lock(&matrix_dev->mdevs_lock);
-
-	/* If a KVM guest is running, disallow unassignment of control domain */
-	if (matrix_mdev->kvm) {
-		ret = -EBUSY;
-		goto done;
-	}
+	vfio_ap_mdev_get_locks(matrix_mdev);
 
 	ret = kstrtoul(buf, 0, &domid);
 	if (ret)
 		goto done;
-	if (domid > max_domid) {
+
+	if (domid > matrix_mdev->matrix.apm_max) {
 		ret = -ENODEV;
 		goto done;
 	}
 
 	clear_bit_inv(domid, matrix_mdev->matrix.adm);
 
-	if (test_bit_inv(domid, matrix_mdev->shadow_apcb.adm))
+	if (test_bit_inv(domid, matrix_mdev->shadow_apcb.adm)) {
 		clear_bit_inv(domid, matrix_mdev->shadow_apcb.adm);
+		vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+	}
 
 	ret = count;
 done:
-	mutex_unlock(&matrix_dev->mdevs_lock);
+	vfio_ap_mdev_put_locks(matrix_mdev);
+
 	return ret;
 }
 static DEVICE_ATTR_WO(unassign_control_domain);
Signature · 1 line
2.31.1
author maintainer
#35 lore raw

PATCH v18 14/18 s390/vfio-ap: sysfs attribute to display the guest's matrix

In reply to Tony Krowiak

The matrix of adapters and domains configured in a guest's APCB may differ from the matrix of adapters and domains assigned to the matrix mdev, so this patch introduces a sysfs attribute to display the matrix of adapters and domains that are or will be assigned to the APCB of a guest that is or will be using the matrix mdev. For a matrix mdev denoted by $uuid, the guest matrix can be displayed as follows:

cat /sys/devices/vfio_ap/matrix/$uuid/guest_matrix

---
 drivers/s390/crypto/vfio_ap_ops.c | 50 +++++++++++++++++++++++--------
 1 file changed, 37 insertions(+), 13 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +37 -13
index 63dfb9b89581..02f75c4207b0 100644
@@ -1170,28 +1170,24 @@ static ssize_t control_domains_show(struct device *dev,
 }
 static DEVICE_ATTR_RO(control_domains);
 
-static ssize_t matrix_show(struct device *dev, struct device_attribute *attr,
-			   char *buf)
+static ssize_t vfio_ap_mdev_matrix_show(struct ap_matrix *matrix, char *buf)
 {
-	struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev);
 	char *bufpos = buf;
 	unsigned long apid;
 	unsigned long apqi;
 	unsigned long apid1;
 	unsigned long apqi1;
-	unsigned long napm_bits = matrix_mdev->matrix.apm_max + 1;
-	unsigned long naqm_bits = matrix_mdev->matrix.aqm_max + 1;
+	unsigned long napm_bits = matrix->apm_max + 1;
+	unsigned long naqm_bits = matrix->aqm_max + 1;
 	int nchars = 0;
 	int n;
 
-	apid1 = find_first_bit_inv(matrix_mdev->matrix.apm, napm_bits);
-	apqi1 = find_first_bit_inv(matrix_mdev->matrix.aqm, naqm_bits);
-
-	mutex_lock(&matrix_dev->mdevs_lock);
+	apid1 = find_first_bit_inv(matrix->apm, napm_bits);
+	apqi1 = find_first_bit_inv(matrix->aqm, naqm_bits);
 
 	if ((apid1 < napm_bits) && (apqi1 < naqm_bits)) {
-		for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, napm_bits) {
-			for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm,
+		for_each_set_bit_inv(apid, matrix->apm, napm_bits) {
+			for_each_set_bit_inv(apqi, matrix->aqm,
 					     naqm_bits) {
 				n = sprintf(bufpos, "%02lx.%04lx\n", apid,
 					    apqi);
@@ -1200,25 +1196,52 @@ static ssize_t matrix_show(struct device *dev, struct device_attribute *attr,
 			}
 		}
 	} else if (apid1 < napm_bits) {
-		for_each_set_bit_inv(apid, matrix_mdev->matrix.apm, napm_bits) {
+		for_each_set_bit_inv(apid, matrix->apm, napm_bits) {
 			n = sprintf(bufpos, "%02lx.\n", apid);
 			bufpos += n;
 			nchars += n;
 		}
 	} else if (apqi1 < naqm_bits) {
-		for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, naqm_bits) {
+		for_each_set_bit_inv(apqi, matrix->aqm, naqm_bits) {
 			n = sprintf(bufpos, ".%04lx\n", apqi);
 			bufpos += n;
 			nchars += n;
 		}
 	}
 
+	return nchars;
+}
+
+static ssize_t matrix_show(struct device *dev, struct device_attribute *attr,
+			   char *buf)
+{
+	ssize_t nchars;
+	struct mdev_device *mdev = mdev_from_dev(dev);
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+	nchars = vfio_ap_mdev_matrix_show(&matrix_mdev->matrix, buf);
 	mutex_unlock(&matrix_dev->mdevs_lock);
 
 	return nchars;
 }
 static DEVICE_ATTR_RO(matrix);
 
+static ssize_t guest_matrix_show(struct device *dev,
+				 struct device_attribute *attr, char *buf)
+{
+	ssize_t nchars;
+	struct mdev_device *mdev = mdev_from_dev(dev);
+	struct ap_matrix_mdev *matrix_mdev = mdev_get_drvdata(mdev);
+
+	mutex_lock(&matrix_dev->mdevs_lock);
+	nchars = vfio_ap_mdev_matrix_show(&matrix_mdev->shadow_apcb, buf);
+	mutex_unlock(&matrix_dev->mdevs_lock);
+
+	return nchars;
+}
+static DEVICE_ATTR_RO(guest_matrix);
+
 static struct attribute *vfio_ap_mdev_attrs[] = {
 	&dev_attr_assign_adapter.attr,
 	&dev_attr_unassign_adapter.attr,
@@ -1228,6 +1251,7 @@ static struct attribute *vfio_ap_mdev_attrs[] = {
 	&dev_attr_unassign_control_domain.attr,
 	&dev_attr_control_domains.attr,
 	&dev_attr_matrix.attr,
+	&dev_attr_guest_matrix.attr,
 	NULL,
 };
 
Signature · 1 line
2.31.1
author maintainer
#41 lore raw

PATCH v18 16/18 s390/vfio-ap: update docs to include dynamic config support

In reply to Tony Krowiak

Update the documentation in vfio-ap.rst to include information about the AP dynamic configuration support (e.g., hot plug of adapters, domains and control domains via the matrix mediated device's sysfs assignment attributes). This patch also makes a few minor tweaks to make corrections and clarifications.

---
 Documentation/s390/vfio-ap.rst | 492 +++++++++++++++++++++++----------
 1 file changed, 346 insertions(+), 146 deletions(-)
Documentation/s390/vfio-ap.rst +346 -146
index f57ae621f33e..f4b8748ab9a8 100644
@@ -123,27 +123,24 @@ Let's now take a look at how AP instructions executed on a guest are interpreted
 by the hardware.
 
 A satellite control block called the Crypto Control Block (CRYCB) is attached to
-our main hardware virtualization control block. The CRYCB contains three fields
-to identify the adapters, usage domains and control domains assigned to the KVM
-guest:
+our main hardware virtualization control block. The CRYCB contains an AP Control
+Block (APCB) that has three fields to identify the adapters, usage domains and
+control domains assigned to the KVM guest:
 
 * The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
-  to the KVM guest. Each bit in the mask, from left to right (i.e. from most
-  significant to least significant bit in big endian order), corresponds to
+  to the KVM guest. Each bit in the mask, from left to right, corresponds to
   an APID from 0-255. If a bit is set, the corresponding adapter is valid for
   use by the KVM guest.
 
 * The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains
-  assigned to the KVM guest. Each bit in the mask, from left to right (i.e. from
-  most significant to least significant bit in big endian order), corresponds to
-  an AP queue index (APQI) from 0-255. If a bit is set, the corresponding queue
-  is valid for use by the KVM guest.
+  assigned to the KVM guest. Each bit in the mask, from left to right,
+  corresponds to an AP queue index (APQI) from 0-255. If a bit is set, the
+  corresponding queue is valid for use by the KVM guest.
 
 * The AP Domain Mask field is a bit mask that identifies the AP control domains
   assigned to the KVM guest. The ADM bit mask controls which domains can be
   changed by an AP command-request message sent to a usage domain from the
-  guest. Each bit in the mask, from left to right (i.e. from most significant to
-  least significant bit in big endian order), corresponds to a domain from
+  guest. Each bit in the mask, from left to right, corresponds to a domain from
   0-255. If a bit is set, the corresponding domain can be modified by an AP
   command-request message sent to a usage domain.
 
@@ -151,10 +148,10 @@ If you recall from the description of an AP Queue, AP instructions include
 an APQN to identify the AP queue to which an AP command-request message is to be
 sent (NQAP and PQAP instructions), or from which a command-reply message is to
 be received (DQAP instruction). The validity of an APQN is defined by the matrix
-calculated from the APM and AQM; it is the cross product of all assigned adapter
-numbers (APM) with all assigned queue indexes (AQM). For example, if adapters 1
-and 2 and usage domains 5 and 6 are assigned to a guest, the APQNs (1,5), (1,6),
-(2,5) and (2,6) will be valid for the guest.
+calculated from the APM and AQM; it is the Cartesian product of all assigned
+adapter numbers (APM) with all assigned queue indexes (AQM). For example, if
+adapters 1 and 2 and usage domains 5 and 6 are assigned to a guest, the APQNs
+(1,5), (1,6), (2,5) and (2,6) will be valid for the guest.
 
 The APQNs can provide secure key functionality - i.e., a private key is stored
 on the adapter card for each of its domains - so each APQN must be assigned to
@@ -192,7 +189,7 @@ The design introduces three new objects:
 
 1. AP matrix device
 2. VFIO AP device driver (vfio_ap.ko)
-3. VFIO AP mediated matrix pass-through device
+3. VFIO AP mediated pass-through device
 
 The VFIO AP device driver
 -------------------------
@@ -200,12 +197,13 @@ The VFIO AP (vfio_ap) device driver serves the following purposes:
 
 1. Provides the interfaces to secure APQNs for exclusive use of KVM guests.
 
-2. Sets up the VFIO mediated device interfaces to manage a mediated matrix
+2. Sets up the VFIO mediated device interfaces to manage a vfio_ap mediated
    device and creates the sysfs interfaces for assigning adapters, usage
    domains, and control domains comprising the matrix for a KVM guest.
 
-3. Configures the APM, AQM and ADM in the CRYCB referenced by a KVM guest's
-   SIE state description to grant the guest access to a matrix of AP devices
+3. Configures the APM, AQM and ADM in the APCB contained in the CRYCB referenced
+   by a KVM guest's SIE state description to grant the guest access to a matrix
+   of AP devices
 
 Reserve APQNs for exclusive use of KVM guests
 ---------------------------------------------
@@ -235,10 +233,10 @@ reserved::
   |                  |       8 probe        |                  |
   +--------^---------+                      +--^--^------------+
   6 edit   |                                   |  |
-    apmask |     +-----------------------------+  | 9 mdev create
+    apmask |     +-----------------------------+  | 11 mdev create
     aqmask |     |           1 modprobe           |
   +--------+-----+---+           +----------------+-+         +----------------+
-  |                  |           |                  |8 create |     mediated   |
+  |                  |           |                  |10 create|     mediated   |
   |      admin       |           | VFIO device core |--------->     matrix     |
   |                  +           |                  |         |     device     |
   +------+-+---------+           +--------^---------+         +--------^-------+
@@ -246,14 +244,14 @@ reserved::
 	 | | 9 create vfio_ap-passthrough |                            |
 	 | +------------------------------+                            |
 	 +-------------------------------------------------------------+
-		     10  assign adapter/domain/control domain
+		     12  assign adapter/domain/control domain
 
 The process for reserving an AP queue for use by a KVM guest is:
 
 1. The administrator loads the vfio_ap device driver
 2. The vfio-ap driver during its initialization will register a single 'matrix'
    device with the device core. This will serve as the parent device for
-   all mediated matrix devices used to configure an AP matrix for a guest.
+   all vfio_ap mediated devices used to configure an AP matrix for a guest.
 3. The /sys/devices/vfio_ap/matrix device is created by the device core
 4. The vfio_ap device driver will register with the AP bus for AP queue devices
    of type 10 and higher (CEX4 and newer). The driver will provide the vfio_ap
@@ -269,24 +267,24 @@ The process for reserving an AP queue for use by a KVM guest is:
    default zcrypt cex4queue driver.
 8. The AP bus probes the vfio_ap device driver to bind the queues reserved for
    it.
-9. The administrator creates a passthrough type mediated matrix device to be
+9. The administrator creates a passthrough type vfio_ap mediated device to be
    used by a guest
 10. The administrator assigns the adapters, usage domains and control domains
     to be exclusively used by a guest.
 
 Set up the VFIO mediated device interfaces
 ------------------------------------------
-The VFIO AP device driver utilizes the common interface of the VFIO mediated
+The VFIO AP device driver utilizes the common interfaces of the VFIO mediated
 device core driver to:
 
-* Register an AP mediated bus driver to add a mediated matrix device to and
+* Register an AP mediated bus driver to add a vfio_ap mediated device to and
   remove it from a VFIO group.
-* Create and destroy a mediated matrix device
-* Add a mediated matrix device to and remove it from the AP mediated bus driver
-* Add a mediated matrix device to and remove it from an IOMMU group
+* Create and destroy a vfio_ap mediated device
+* Add a vfio_ap mediated device to and remove it from the AP mediated bus driver
+* Add a vfio_ap mediated device to and remove it from an IOMMU group
 
 The following high-level block diagram shows the main components and interfaces
-of the VFIO AP mediated matrix device driver::
+of the VFIO AP mediated device driver::
 
    +-------------+
    |             |
@@ -343,7 +341,7 @@ matrix device.
 	* device_api:
 	    the mediated device type's API
 	* available_instances:
-	    the number of mediated matrix passthrough devices
+	    the number of vfio_ap mediated passthrough devices
 	    that can be created
 	* device_api:
 	    specifies the VFIO API
@@ -351,29 +349,37 @@ matrix device.
     This attribute group identifies the user-defined sysfs attributes of the
     mediated device. When a device is registered with the VFIO mediated device
     framework, the sysfs attribute files identified in the 'mdev_attr_groups'
-    structure will be created in the mediated matrix device's directory. The
-    sysfs attributes for a mediated matrix device are:
+    structure will be created in the vfio_ap mediated device's directory. The
+    sysfs attributes for a vfio_ap mediated device are:
 
     assign_adapter / unassign_adapter:
       Write-only attributes for assigning/unassigning an AP adapter to/from the
-      mediated matrix device. To assign/unassign an adapter, the APID of the
-      adapter is echoed to the respective attribute file.
+      vfio_ap mediated device. To assign/unassign an adapter, the APID of the
+      adapter is echoed into the respective attribute file.
     assign_domain / unassign_domain:
       Write-only attributes for assigning/unassigning an AP usage domain to/from
-      the mediated matrix device. To assign/unassign a domain, the domain
-      number of the usage domain is echoed to the respective attribute
+      the vfio_ap mediated device. To assign/unassign a domain, the domain
+      number of the usage domain is echoed into the respective attribute
       file.
     matrix:
-      A read-only file for displaying the APQNs derived from the cross product
-      of the adapter and domain numbers assigned to the mediated matrix device.
+      A read-only file for displaying the APQNs derived from the Cartesian
+      product of the adapter and domain numbers assigned to the vfio_ap mediated
+      device.
+    guest_matrix:
+      A read-only file for displaying the APQNs derived from the Cartesian
+      product of the adapter and domain numbers assigned to the APM and AQM
+      fields respectively of the KVM guest's CRYCB. This may differ from the
+      the APQNs assigned to the vfio_ap mediated device if any APQN does not
+      reference a queue device bound to the vfio_ap device driver (i.e., the
+      queue is not in the host's AP configuration).
     assign_control_domain / unassign_control_domain:
       Write-only attributes for assigning/unassigning an AP control domain
-      to/from the mediated matrix device. To assign/unassign a control domain,
-      the ID of the domain to be assigned/unassigned is echoed to the respective
-      attribute file.
+      to/from the vfio_ap mediated device. To assign/unassign a control domain,
+      the ID of the domain to be assigned/unassigned is echoed into the
+      respective attribute file.
     control_domains:
       A read-only file for displaying the control domain numbers assigned to the
-      mediated matrix device.
+      vfio_ap mediated device.
 
 * functions:
 
@@ -383,45 +389,75 @@ matrix device.
     * Store the reference to the KVM structure for the guest using the mdev
     * Store the AP matrix configuration for the adapters, domains, and control
       domains assigned via the corresponding sysfs attributes files
+    * Store the AP matrix configuration for the adapters, domains and control
+      domains available to a guest. A guest may not be provided access to APQNs
+      referencing queue devices that do not exist, or are not bound to the
+      vfio_ap device driver.
 
   remove:
-    deallocates the mediated matrix device's ap_matrix_mdev structure. This will
-    be allowed only if a running guest is not using the mdev.
+    deallocates the vfio_ap mediated device's ap_matrix_mdev structure.
+    This will be allowed only if a running guest is not using the mdev.
 
 * callback interfaces
 
-  open:
+  open_device:
     The vfio_ap driver uses this callback to register a
-    VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the mdev matrix
-    device. The open is invoked when QEMU connects the VFIO iommu group
-    for the mdev matrix device to the MDEV bus. Access to the KVM structure used
-    to configure the KVM guest is provided via this callback. The KVM structure,
-    is used to configure the guest's access to the AP matrix defined via the
-    mediated matrix device's sysfs attribute files.
-  release:
+    VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the matrix mdev
+    devices. The open_device callback is invoked by userspace to connect the
+    VFIO iommu group for the matrix mdev device to the MDEV bus. Access to the
+    KVM structure used to configure the KVM guest is provided via this callback.
+    The KVM structure, is used to configure the guest's access to the AP matrix
+    defined via the vfio_ap mediated device's sysfs attribute files.
+
+  close_device:
     unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the
-    mdev matrix device and deconfigures the guest's AP matrix.
+    matrix mdev device and deconfigures the guest's AP matrix.
 
-Configure the APM, AQM and ADM in the CRYCB
--------------------------------------------
-Configuring the AP matrix for a KVM guest will be performed when the
+  ioctl:
+    this callback handles the VFIO_DEVICE_GET_INFO and VFIO_DEVICE_RESET ioctls
+    defined by the vfio framework.
+
+Configure the guest's AP resources
+----------------------------------
+Configuring the AP resources for a KVM guest will be performed when the
 VFIO_GROUP_NOTIFY_SET_KVM notifier callback is invoked. The notifier
-function is called when QEMU connects to KVM. The guest's AP matrix is
-configured via it's CRYCB by:
+function is called when userspace connects to KVM. The guest's AP resources are
+configured via it's APCB by:
 
 * Setting the bits in the APM corresponding to the APIDs assigned to the
-  mediated matrix device via its 'assign_adapter' interface.
+  vfio_ap mediated device via its 'assign_adapter' interface.
 * Setting the bits in the AQM corresponding to the domains assigned to the
-  mediated matrix device via its 'assign_domain' interface.
+  vfio_ap mediated device via its 'assign_domain' interface.
 * Setting the bits in the ADM corresponding to the domain dIDs assigned to the
-  mediated matrix device via its 'assign_control_domains' interface.
+  vfio_ap mediated device via its 'assign_control_domains' interface.
+
+The linux device model precludes passing a device through to a KVM guest that
+is not bound to the device driver facilitating its pass-through. Consequently,
+an APQN that does not reference a queue device bound to the vfio_ap device
+driver will not be assigned to a KVM guest's matrix. The AP architecture,
+however, does not provide a means to filter individual APQNs from the guest's
+matrix, so the adapters, domains and control domains assigned to vfio_ap
+mediated device via its sysfs 'assign_adapter', 'assign_domain' and
+'assign_control_domain' interfaces will be filtered before providing the AP
+configuration to a guest:
+
+* The APIDs of the adapters, the APQIs of the domains and the domain numbers of
+  the control domains assigned to the matrix mdev that are not also assigned to
+  the host's AP configuration will be filtered.
+
+* Each APQN derived from the Cartesian product of the APIDs and APQIs assigned
+  to the vfio_ap mdev is examined and if any one of them does not reference a
+  queue device bound to the vfio_ap device driver, the adapter will not be
+  plugged into the guest (i.e., the bit corresponding to its APID will not be
+  set in the APM of the guest's APCB).
 
 The CPU model features for AP
 -----------------------------
-The AP stack relies on the presence of the AP instructions as well as two
-facilities: The AP Facilities Test (APFT) facility; and the AP Query
-Configuration Information (QCI) facility. These features/facilities are made
-available to a KVM guest via the following CPU model features:
+The AP stack relies on the presence of the AP instructions as well as three
+facilities: The AP Facilities Test (APFT) facility; the AP Query
+Configuration Information (QCI) facility; and the AP Queue Interruption Control
+facility. These features/facilities are made available to a KVM guest via the
+following CPU model features:
 
 1. ap: Indicates whether the AP instructions are installed on the guest. This
    feature will be enabled by KVM only if the AP instructions are installed
@@ -435,24 +471,28 @@ available to a KVM guest via the following CPU model features:
    can be made available to the guest only if it is available on the host (i.e.,
    facility bit 12 is set).
 
+4. apqi: Indicates AP Queue Interruption Control faclity is available on the
+   guest. This facility can be made available to the guest only if it is
+   available on the host (i.e., facility bit 65 is set).
+
 Note: If the user chooses to specify a CPU model different than the 'host'
 model to QEMU, the CPU model features and facilities need to be turned on
 explicitly; for example::
 
-     /usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on
+     /usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on,apqi=on
 
 A guest can be precluded from using AP features/facilities by turning them off
 explicitly; for example::
 
-     /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off
+     /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off,apqi=off
 
 Note: If the APFT facility is turned off (apft=off) for the guest, the guest
-will not see any AP devices. The zcrypt device drivers that register for type 10
-and newer AP devices - i.e., the cex4card and cex4queue device drivers - need
-the APFT facility to ascertain the facilities installed on a given AP device. If
-the APFT facility is not installed on the guest, then the probe of device
-drivers will fail since only type 10 and newer devices can be configured for
-guest use.
+will not see any AP devices. The zcrypt device drivers on the guest that
+register for type 10 and newer AP devices - i.e., the cex4card and cex4queue
+device drivers - need the APFT facility to ascertain the facilities installed on
+a given AP device. If the APFT facility is not installed on the guest, then no
+adapter or domain devices will get created by the AP bus running on the
+guest because only type 10 and newer devices can be configured for guest use.
 
 Example
 =======
@@ -471,7 +511,7 @@ CARD.DOMAIN TYPE  MODE
 05.00ab     CEX5C CCA-Coproc
 06          CEX5A Accelerator
 06.0004     CEX5A Accelerator
-06.00ab     CEX5C CCA-Coproc
+06.00ab     CEX5A Accelerator
 =========== ===== ============
 
 Guest2
@@ -479,9 +519,9 @@ Guest2
 =========== ===== ============
 CARD.DOMAIN TYPE  MODE
 =========== ===== ============
-05          CEX5A Accelerator
-05.0047     CEX5A Accelerator
-05.00ff     CEX5A Accelerator
+05          CEX5C CCA-Coproc
+05.0047     CEX5C CCA-Coproc
+05.00ff     CEX5C CCA-Coproc
 =========== ===== ============
 
 Guest3
@@ -529,40 +569,56 @@ These are the steps:
 
 2. Secure the AP queues to be used by the three guests so that the host can not
    access them. To secure them, there are two sysfs files that specify
-   bitmasks marking a subset of the APQN range as 'usable by the default AP
-   queue device drivers' or 'not usable by the default device drivers' and thus
-   available for use by the vfio_ap device driver'. The location of the sysfs
-   files containing the masks are::
+   bitmasks marking a subset of the APQN range as usable only by the default AP
+   queue device drivers. All remaining APQNs are available for use by
+   any other device driver. The vfio_ap device driver is currently the only
+   non-default device driver. The location of the sysfs files containing the
+   masks are::
 
      /sys/bus/ap/apmask
      /sys/bus/ap/aqmask
 
    The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
-   (APID). Each bit in the mask, from left to right (i.e., from most significant
-   to least significant bit in big endian order), corresponds to an APID from
-   0-255. If a bit is set, the APID is marked as usable only by the default AP
-   queue device drivers; otherwise, the APID is usable by the vfio_ap
-   device driver.
+   (APID). Each bit in the mask, from left to right, corresponds to an APID from
+   0-255. If a bit is set, the APID belongs to the subset of APQNs marked as
+   available only to the default AP queue device drivers.
 
    The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
-   (APQI). Each bit in the mask, from left to right (i.e., from most significant
-   to least significant bit in big endian order), corresponds to an APQI from
-   0-255. If a bit is set, the APQI is marked as usable only by the default AP
-   queue device drivers; otherwise, the APQI is usable by the vfio_ap device
-   driver.
+   (APQI). Each bit in the mask, from left to right, corresponds to an APQI from
+   0-255. If a bit is set, the APQI belongs to the subset of APQNs marked as
+   available only to the default AP queue device drivers.
+
+   The Cartesian product of the APIDs corresponding to the bits set in the
+   apmask and the APQIs corresponding to the bits set in the aqmask comprise
+   the subset of APQNs that can be used only by the host default device drivers.
+   All other APQNs are available to the non-default device drivers such as the
+   vfio_ap driver.
+
+   Take, for example, the following masks::
+
+      apmask:
+      0x7d00000000000000000000000000000000000000000000000000000000000000
 
-   Take, for example, the following mask::
+      aqmask:
+      0x8000000000000000000000000000000000000000000000000000000000000000
 
-      0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
+   The masks indicate:
 
-    It indicates:
+   * Adapters 1, 2, 3, 4, 5, and 7 are available for use by the host default
+     device drivers.
 
-      1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6
-      belong to the vfio_ap device driver's pool.
+   * Domain 0 is available for use by the host default device drivers
+
+   * The subset of APQNs available for use only by the default host device
+     drivers are:
+
+     (1,0), (2,0), (3,0), (4.0), (5,0) and (7,0)
+
+   * All other APQNs are available for use by the non-default device drivers.
 
    The APQN of each AP queue device assigned to the linux host is checked by the
-   AP bus against the set of APQNs derived from the cross product of APIDs
-   and APQIs marked as usable only by the default AP queue device drivers. If a
+   AP bus against the set of APQNs derived from the Cartesian product of APIDs
+   and APQIs marked as available to the default AP queue device drivers. If a
    match is detected,  only the default AP queue device drivers will be probed;
    otherwise, the vfio_ap device driver will be probed.
 
@@ -579,8 +635,7 @@ These are the steps:
 
 	   0x4100000000000000000000000000000000000000000000000000000000000000
 
-	Keep in mind that the mask reads from left to right (i.e., most
-	significant to least significant bit in big endian order), so the mask
+	Keep in mind that the mask reads from left to right, so the mask
 	above identifies device numbers 1 and 7 (01000001).
 
 	If the string is longer than the mask, the operation is terminated with
@@ -626,11 +681,22 @@ These are the steps:
 	    default drivers pool:    adapter 0-15, domain 1
 	    alternate drivers pool:  adapter 16-255, domains 0, 2-255
 
+   Note ***:
+   Changing a mask such that one or more APQNs will be taken from a vfio_ap
+   mediated device (see below) will fail with an error (EBUSY). A message
+   is logged to the kernel ring buffer which can be viewed with the 'dmesg'
+   command. The output identifies each APQN flagged as 'in use' and identifies
+   the vfio_ap mediated device to which it is assigned; for example:
+
+   Userspace may not re-assign queue 05.0054 already assigned to 62177883-f1bb-47f0-914d-32a22e3a8804
+   Userspace may not re-assign queue 04.0054 already assigned to cef03c3c-903d-4ecc-9a83-40694cb8aee4
+
 Securing the APQNs for our example
 ----------------------------------
    To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047,
    06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding
-   APQNs can either be removed from the default masks::
+   APQNs can be removed from the default masks using either of the following
+   commands::
 
       echo -5,-6 > /sys/bus/ap/apmask
 
@@ -683,7 +749,7 @@ Securing the APQNs for our example
 
      /sys/devices/vfio_ap/matrix/
      --- [mdev_supported_types]
-     ------ [vfio_ap-passthrough] (passthrough mediated matrix device type)
+     ------ [vfio_ap-passthrough] (passthrough vfio_ap mediated device type)
      --------- create
      --------- [devices]
 
@@ -734,6 +800,9 @@ Securing the APQNs for our example
      ----------------unassign_control_domain
      ----------------unassign_domain
 
+   Note *****: The vfio_ap mdevs do not persist across reboots unless the
+               mdevctl tool is used to create and persist them.
+
 4. The administrator now needs to configure the matrixes for the mediated
    devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3).
 
@@ -755,6 +824,10 @@ Securing the APQNs for our example
 
 	 cat matrix
 
+   To display the matrix that is or will be assigned to Guest1::
+
+	 cat guest_matrix
+
    This is how the matrix is configured for Guest2::
 
       echo 5 > assign_adapter
@@ -774,17 +847,24 @@ Securing the APQNs for our example
      higher than the maximum is specified, the operation will terminate with
      an error (ENODEV).
 
-   * All APQNs that can be derived from the adapter ID and the IDs of
-     the previously assigned domains must be bound to the vfio_ap device
-     driver. If no domains have yet been assigned, then there must be at least
-     one APQN with the specified APID bound to the vfio_ap driver. If no such
-     APQNs are bound to the driver, the operation will terminate with an
-     error (EADDRNOTAVAIL).
+     Note: The maximum adapter number can be obtained via the sysfs
+	   /sys/bus/ap/ap_max_adapter_id attribute file.
+
+   * Each APQN derived from the Cartesian product of the APID of the adapter
+     being assigned and the APQIs of the domains previously assigned:
 
-     No APQN that can be derived from the adapter ID and the IDs of the
-     previously assigned domains can be assigned to another mediated matrix
-     device. If an APQN is assigned to another mediated matrix device, the
-     operation will terminate with an error (EADDRINUSE).
+     - Must only be available to the vfio_ap device driver as specified in the
+       sysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even
+       one APQN is reserved for use by the host device driver, the operation
+       will terminate with an error (EADDRNOTAVAIL).
+
+     - Must NOT be assigned to another vfio_ap mediated device. If even one APQN
+       is assigned to another vfio_ap mediated device, the operation will
+       terminate with an error (EBUSY).
+
+     - Must NOT be assigned while the sysfs /sys/bus/ap/apmask and
+       sys/bus/ap/aqmask attribute files are being edited or the operation may
+       terminate with an error (EBUSY).
 
    In order to successfully assign a domain:
 
@@ -793,41 +873,50 @@ Securing the APQNs for our example
      higher than the maximum is specified, the operation will terminate with
      an error (ENODEV).
 
-   * All APQNs that can be derived from the domain ID and the IDs of
-     the previously assigned adapters must be bound to the vfio_ap device
-     driver. If no domains have yet been assigned, then there must be at least
-     one APQN with the specified APQI bound to the vfio_ap driver. If no such
-     APQNs are bound to the driver, the operation will terminate with an
-     error (EADDRNOTAVAIL).
+     Note: The maximum domain number can be obtained via the sysfs
+	   /sys/bus/ap/ap_max_domain_id attribute file.
+
+    * Each APQN derived from the Cartesian product of the APQI of the domain
+      being assigned and the APIDs of the adapters previously assigned:
 
-     No APQN that can be derived from the domain ID and the IDs of the
-     previously assigned adapters can be assigned to another mediated matrix
-     device. If an APQN is assigned to another mediated matrix device, the
-     operation will terminate with an error (EADDRINUSE).
+     - Must only be available to the vfio_ap device driver as specified in the
+       sysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even
+       one APQN is reserved for use by the host device driver, the operation
+       will terminate with an error (EADDRNOTAVAIL).
 
-   In order to successfully assign a control domain, the domain number
-   specified must represent a value from 0 up to the maximum domain number
-   configured for the system. If a control domain number higher than the maximum
-   is specified, the operation will terminate with an error (ENODEV).
+     - Must NOT be assigned to another vfio_ap mediated device. If even one APQN
+       is assigned to another vfio_ap mediated device, the operation will
+       terminate with an error (EBUSY).
+
+     - Must NOT be assigned while the sysfs /sys/bus/ap/apmask and
+       sys/bus/ap/aqmask attribute files are being edited or the operation may
+       terminate with an error (EBUSY).
+
+   In order to successfully assign a control domain:
+
+   * The domain number specified must represent a value from 0 up to the maximum
+     domain number configured for the system. If a control domain number higher
+     than the maximum is specified, the operation will terminate with an
+     error (ENODEV).
 
 5. Start Guest1::
 
-     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
+     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
 	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
 
 7. Start Guest2::
 
-     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
+     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
 	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...
 
 7. Start Guest3::
 
-     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \
+     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
 	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ...
 
-When the guest is shut down, the mediated matrix devices may be removed.
+When the guest is shut down, the vfio_ap mediated devices may be removed.
 
-Using our example again, to remove the mediated matrix device $uuid1::
+Using our example again, to remove the vfio_ap mediated device $uuid1::
 
    /sys/devices/vfio_ap/matrix/
       --- [mdev_supported_types]
@@ -840,26 +929,137 @@ Using our example again, to remove the mediated matrix device $uuid1::
 
    echo 1 > remove
 
-This will remove all of the mdev matrix device's sysfs structures including
-the mdev device itself. To recreate and reconfigure the mdev matrix device,
+This will remove all of the matrix mdev device's sysfs structures including
+the mdev device itself. To recreate and reconfigure the matrix mdev device,
 all of the steps starting with step 3 will have to be performed again. Note
-that the remove will fail if a guest using the mdev is still running.
+that the remove will fail if a guest using the vfio_ap mdev is still running.
 
-It is not necessary to remove an mdev matrix device, but one may want to
+It is not necessary to remove a vfio_ap mdev, but one may want to
 remove it if no guest will use it during the remaining lifetime of the linux
-host. If the mdev matrix device is removed, one may want to also reconfigure
+host. If the vfio_ap mdev is removed, one may want to also reconfigure
 the pool of adapters and queues reserved for use by the default drivers.
 
+Hot plug/unplug support:
+================
+An adapter, domain or control domain may be hot plugged into a running KVM
+guest by assigning it to the vfio_ap mediated device being used by the guest if
+the following conditions are met:
+
+* The adapter, domain or control domain must also be assigned to the host's
+  AP configuration.
+
+* Each APQN derived from the Cartesian product comprised of the APID of the
+  adapter being assigned and the APQIs of the domains assigned must reference a
+  queue device bound to the vfio_ap device driver.
+
+* To hot plug a domain, each APQN derived from the Cartesian product
+  comprised of the APQI of the domain being assigned and the APIDs of the
+  adapters assigned must reference a queue device bound to the vfio_ap device
+  driver.
+
+An adapter, domain or control domain may be hot unplugged from a running KVM
+guest by unassigning it from the vfio_ap mediated device being used by the
+guest.
+
+Over-provisioning of AP queues for a KVM guest:
+==============================================
+Over-provisioning is defined herein as the assignment of adapters or domains to
+a vfio_ap mediated device that do not reference AP devices in the host's AP
+configuration. The idea here is that when the adapter or domain becomes
+available, it will be automatically hot-plugged into the KVM guest using
+the vfio_ap mediated device to which it is assigned as long as each new APQN
+resulting from plugging it in references a queue device bound to the vfio_ap
+device driver.
+
 Limitations
 ===========
-* The KVM/kernel interfaces do not provide a way to prevent restoring an APQN
-  to the default drivers pool of a queue that is still assigned to a mediated
-  device in use by a guest. It is incumbent upon the administrator to
-  ensure there is no mediated device in use by a guest to which the APQN is
-  assigned lest the host be given access to the private data of the AP queue
-  device such as a private key configured specifically for the guest.
+Live guest migration is not supported for guests using AP devices without
+intervention by a system administrator. Before a KVM guest can be migrated,
+the vfio_ap mediated device must be removed. Unfortunately, it can not be
+removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while
+the mdev is in use by a KVM guest. If the guest is being emulated by QEMU,
+its mdev can be hot unplugged from the guest in one of two ways:
+
+1. If the KVM guest was started with libvirt, you can hot unplug the mdev via
+   the following commands:
+
+      virsh detach-device <guestname> <path-to-device-xml>
+
+      For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from
+      the guest named 'my-guest':
+
+         virsh detach-device my-guest ~/config/my-guest-hostdev.xml
+
+            The contents of my-guest-hostdev.xml:
+
+            <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
+              <source>
+                <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
+              </source>
+            </hostdev>
+
+
+      virsh qemu-monitor-command <guest-name> --hmp "device-del <device-id>"
+
+      For example, to hot unplug the vfio_ap mediated device identified on the
+      qemu command line with 'id=hostdev0' from the guest named 'my-guest':
+
+         virsh qemu-monitor-command my-guest --hmp "device_del hostdev0"
+
+2. A vfio_ap mediated device can be hot unplugged by attaching the qemu monitor
+   to the guest and using the following qemu monitor command:
+
+      (QEMU) device-del id=<device-id>
+
+      For example, to hot unplug the vfio_ap mediated device that was specified
+      on the qemu command line with 'id=hostdev0' when the guest was started:
+
+         (QEMU) device-del id=hostdev0
+
+After live migration of the KVM guest completes, an AP configuration can be
+restored to the KVM guest by hot plugging a vfio_ap mediated device on the target
+system into the guest in one of two ways:
+
+1. If the KVM guest was started with libvirt, you can hot plug a matrix mediated
+   device into the guest via the following virsh commands:
+
+   virsh attach-device <guestname> <path-to-device-xml>
+
+      For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into
+      the guest named 'my-guest':
+
+         virsh attach-device my-guest ~/config/my-guest-hostdev.xml
+
+            The contents of my-guest-hostdev.xml:
+
+            <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
+              <source>
+                <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
+              </source>
+            </hostdev>
+
+
+   virsh qemu-monitor-command <guest-name> --hmp \
+   "device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
+
+      For example, to hot plug the vfio_ap mediated device
+      62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with
+      device-id hostdev0:
+
+      virsh qemu-monitor-command my-guest --hmp \
+      "device_add vfio-ap,\
+      sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
+      id=hostdev0"
+
+2. A vfio_ap mediated device can be hot plugged by attaching the qemu monitor
+   to the guest and using the following qemu monitor command:
+
+      (qemu) device_add "vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
 
-* Dynamically modifying the AP matrix for a running guest (which would amount to
-  hot(un)plug of AP devices for the guest) is currently not supported
+      For example, to plug the vfio_ap mediated device
+      62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id
+      hostdev0:
 
-* Live guest migration is not supported for guests using AP devices.
+         (QEMU) device-add "vfio-ap,\
+         sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
+         id=hostdev0"
Signature · 1 line
2.31.1
author maintainer
#46 lore raw

PATCH v18 18/18 MAINTAINERS: pick up all vfio_ap docs for VFIO AP maintainers

In reply to Tony Krowiak

A new document, Documentation/s390/vfio-ap-locking.rst was added. Make sure the new document is picked up for the VFIO AP maintainers by using a
wildcard: Documentation/s390/vfio-ap*.

---
 MAINTAINERS | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
MAINTAINERS +1 -1
index fca970a46e77..ece0968f9fa4 100644
@@ -16908,7 +16908,7 @@ M:	Jason Herne <[email protected]>
 L:	[email protected]
 S:	Supported
 W:	http://www.ibm.com/developerworks/linux/linux390/
-F:	Documentation/s390/vfio-ap.rst
+F:	Documentation/s390/vfio-ap*
 F:	drivers/s390/crypto/vfio_ap_drv.c
 F:	drivers/s390/crypto/vfio_ap_ops.c
 F:	drivers/s390/crypto/vfio_ap_private.h
Signature · 1 line
2.31.1
author maintainer
#47 lore raw
In reply to Tony Krowiak

PING!!

Quoted 174 lines · Tony Krowiak

The current design for AP pass-through does not support making dynamic changes to the AP matrix of a running guest resulting in a few deficiencies this patch series is intended to mitigate:

1. Adapters, domains and control domains can not be added to or removed from a running guest. In order to modify a guest's AP configuration, the guest must be terminated; only then can AP resources be assigned to or unassigned from the guest's matrix mdev. The new AP configuration becomes available to the guest when it is subsequently restarted.

2. The AP bus's /sys/bus/ap/apmask and /sys/bus/ap/aqmask interfaces can be modified by a root user without any restrictions. A change to either mask can result in AP queue devices being unbound from the vfio_ap device driver and bound to a zcrypt device driver even if a guest is using the queues, thus giving the host access to the guest's private crypto data and vice versa.

3. The APQNs derived from the Cartesian product of the APIDs of the adapters and APQIs of the domains assigned to a matrix mdev must reference an AP queue device bound to the vfio_ap device driver. The AP architecture allows assignment of AP resources that are not available to the system, so this artificial restriction is not compliant with the architecture.

4. The AP configuration profile can be dynamically changed for the linux host after a KVM guest is started. For example, a new domain can be dynamically added to the configuration profile via the SE or an HMC connected to a DPM enabled lpar. Likewise, AP adapters can be dynamically configured (online state) and deconfigured (standby state) using the SE, an SCLP command or an HMC connected to a DPM enabled lpar. This can result in inadvertent sharing of AP queues between the guest and host.

5. A root user can manually unbind an AP queue device representing a queue in use by a KVM guest via the vfio_ap device driver's sysfs unbind attribute. In this case, the guest will be using a queue that is not bound to the driver which violates the device model.

This patch series introduces the following changes to the current design to alleviate the shortcomings described above as well as to implement more of the AP architecture:

1. A root user will be prevented from making edits to the AP bus's /sys/bus/ap/apmask or /sys/bus/ap/aqmask if the change would transfer ownership of an APQN from the vfio_ap device driver to a zcrypt driver while the APQN is assigned to a matrix mdev.

2. Allow a root user to hot plug/unplug AP adapters, domains and control domains for a KVM guest using the matrix mdev via its sysfs assign/unassign attributes.

4. Allow assignment of an AP adapter or domain to a matrix mdev even if it results in assignment of an APQN that does not reference an AP queue device bound to the vfio_ap device driver, as long as the APQN is not reserved for use by the default zcrypt drivers (also known as over-provisioning of AP resources). Allowing over-provisioning of AP resources better models the architecture which does not preclude assigning AP resources that are not yet available in the system. Such APQNs, however, will not be assigned to the guest using the matrix mdev; only APQNs referencing AP queue devices bound to the vfio_ap device driver will actually get assigned to the guest.

5. Handle dynamic changes to the AP device model.

1. Rationale for changes to AP bus's apmask/aqmask interfaces:
----------------------------------------------------------
Due to the extremely sensitive nature of cryptographic data, it is imperative that great care be taken to ensure that such data is secured. Allowing a root user, either inadvertently or maliciously, to configure these masks such that a queue is shared between the host and a guest is not only avoidable, it is advisable. It was suggested that this scenario is better handled in user space with management software, but that does not preclude a malicious administrator from using the sysfs interfaces to gain access to a guest's crypto data. It was also suggested that this scenario could be avoided by taking access to the adapter away from the guest and zeroing out the queues prior to the vfio_ap driver releasing the device; however, stealing an adapter in use from a guest as a by-product of an operation is bad and will likely cause problems for the guest unnecessarily. It was decided that the most effective solution with the least number of negative side effects is to prevent the situation at the source.

2. Rationale for hot plug/unplug using matrix mdev sysfs interfaces:
----------------------------------------------------------------
Allowing a user to hot plug/unplug AP resources using the matrix mdev sysfs interfaces circumvents the need to terminate the guest in order to modify its AP configuration. Allowing dynamic configuration makes reconfiguring a guest's AP matrix much less disruptive.

3. Rationale for allowing over-provisioning of AP resources:
-----------------------------------------------------------
Allowing assignment of AP resources to a matrix mdev and ultimately to a guest better models the AP architecture. The architecture does not preclude assignment of unavailable AP resources. If a queue subsequently becomes available while a guest using the matrix mdev to which its APQN is assigned, the guest will be given access to it. If an APQN is dynamically unassigned from the underlying host system, it will automatically become unavailable to the guest.

Change log v17-v18:
------------------
* Added a new document, Documentation/s390/vfio-ap-locking.rst to describe the locking design for this series. This included a patch for adding the new doc to the MAITAINERS file for the VFIO AP maintainers.

* Added Reviewed-by for Halil in patch 6

* Restore filtering in the queue remove callback. (Halil)

* Added patch s390/vfio-ap: hot unplug of AP devices when mdev removed to unplug all AP devices from the guest when the mdev is removed.

* Split patch 9 (s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device) into xx patches:
- allow hot plug/unplug of AP devices when assigned/unassigned

* Replaced v17 patch 08/15, s390/vfio-ap: keep track of active guests with s390/vfio-ap: introduce new mutex to control access to the KVM pointer. No longer tracking guests with a list of struct ap_guest objects. Added a global mutex called kvm_lock which must be held whenever the kvm->lock is taken. (Halil)

* Changed signature of the filtering function to limit the APQNs examined. This change resulted in a cascade of changes to patches 5, 8 and 12. (Halil)

* Removed v17 patch 01/15 (s390/vfio-ap: Set pqap hook when vfio_ap module is loaded). (Halil, Jason).

* Split patch 14 (notify drivers on config changed and scan complete callbacks) into two patches: One to make the AP bus changes and the other to implement the callbacks in the vfio_ap device driver. This was done to facilitate merging the AP bus changes separately from the vfio_ap driver changes. (Harald)

* Removed check of driver flags in the __verify_card_reservations and __verify_queue_reservations functions in ap_bus.c to balance the weighting between the default and vfio_ap drivers. (Harald)

Tony Krowiak (17):
s390/ap: driver callback to indicate resource in use s390/ap: notify drivers on config changed and scan complete callbacks s390/vfio-ap: use new AP bus interface to search for queue devices s390/vfio-ap: move probe and remove callbacks to vfio_ap_ops.c s390/vfio-ap: manage link between queue struct and matrix mdev s390/vfio-ap: introduce shadow APCB
s390/vfio-ap: refresh guest's APCB by filtering APQNs assigned to mdev s390/vfio-ap: allow assignment of unavailable AP queues to mdev device s390/vfio-ap: introduce new mutex to control access to the KVM pointer s390/vfio-ap: allow hot plug/unplug of AP devices when
assigned/unassigned
s390/vfio-ap: hot plug/unplug of AP devices when probed/removed s390/vfio-ap: reset queues after adapter/domain unassignment s390/vfio-ap: implement in-use callback for vfio_ap driver s390/vfio-ap: sysfs attribute to display the guest's matrix s390/vfio-ap: handle config changed and scan complete notification s390/vfio-ap: update docs to include dynamic config support s390/Docs: new doc describing lock usage by the vfio_ap device driver
MAINTAINERS: pick up all vfio_ap docs for VFIO AP maintainers

  Documentation/s390/vfio-ap-locking.rst |  389 +++++++
  Documentation/s390/vfio-ap.rst         |  492 ++++++---
  MAINTAINERS                            |    2 +-
  drivers/s390/crypto/ap_bus.c           |  226 +++-
  drivers/s390/crypto/ap_bus.h           |   16 +
  drivers/s390/crypto/vfio_ap_drv.c      |   71 +-
  drivers/s390/crypto/vfio_ap_ops.c      | 1347 ++++++++++++++++++------
  drivers/s390/crypto/vfio_ap_private.h  |   57 +-
  8 files changed, 2058 insertions(+), 542 deletions(-)

create mode 100644 Documentation/s390/vfio-ap-locking.rst

author maintainer
#48 lore raw
In reply to Tony Krowiak

PING!! Any takers?

Quoted 174 lines · Tony Krowiak

The current design for AP pass-through does not support making dynamic changes to the AP matrix of a running guest resulting in a few deficiencies this patch series is intended to mitigate:

1. Adapters, domains and control domains can not be added to or removed from a running guest. In order to modify a guest's AP configuration, the guest must be terminated; only then can AP resources be assigned to or unassigned from the guest's matrix mdev. The new AP configuration becomes available to the guest when it is subsequently restarted.

2. The AP bus's /sys/bus/ap/apmask and /sys/bus/ap/aqmask interfaces can be modified by a root user without any restrictions. A change to either mask can result in AP queue devices being unbound from the vfio_ap device driver and bound to a zcrypt device driver even if a guest is using the queues, thus giving the host access to the guest's private crypto data and vice versa.

3. The APQNs derived from the Cartesian product of the APIDs of the adapters and APQIs of the domains assigned to a matrix mdev must reference an AP queue device bound to the vfio_ap device driver. The AP architecture allows assignment of AP resources that are not available to the system, so this artificial restriction is not compliant with the architecture.

4. The AP configuration profile can be dynamically changed for the linux host after a KVM guest is started. For example, a new domain can be dynamically added to the configuration profile via the SE or an HMC connected to a DPM enabled lpar. Likewise, AP adapters can be dynamically configured (online state) and deconfigured (standby state) using the SE, an SCLP command or an HMC connected to a DPM enabled lpar. This can result in inadvertent sharing of AP queues between the guest and host.

5. A root user can manually unbind an AP queue device representing a queue in use by a KVM guest via the vfio_ap device driver's sysfs unbind attribute. In this case, the guest will be using a queue that is not bound to the driver which violates the device model.

This patch series introduces the following changes to the current design to alleviate the shortcomings described above as well as to implement more of the AP architecture:

1. A root user will be prevented from making edits to the AP bus's /sys/bus/ap/apmask or /sys/bus/ap/aqmask if the change would transfer ownership of an APQN from the vfio_ap device driver to a zcrypt driver while the APQN is assigned to a matrix mdev.

2. Allow a root user to hot plug/unplug AP adapters, domains and control domains for a KVM guest using the matrix mdev via its sysfs assign/unassign attributes.

4. Allow assignment of an AP adapter or domain to a matrix mdev even if it results in assignment of an APQN that does not reference an AP queue device bound to the vfio_ap device driver, as long as the APQN is not reserved for use by the default zcrypt drivers (also known as over-provisioning of AP resources). Allowing over-provisioning of AP resources better models the architecture which does not preclude assigning AP resources that are not yet available in the system. Such APQNs, however, will not be assigned to the guest using the matrix mdev; only APQNs referencing AP queue devices bound to the vfio_ap device driver will actually get assigned to the guest.

5. Handle dynamic changes to the AP device model.

1. Rationale for changes to AP bus's apmask/aqmask interfaces:
----------------------------------------------------------
Due to the extremely sensitive nature of cryptographic data, it is imperative that great care be taken to ensure that such data is secured. Allowing a root user, either inadvertently or maliciously, to configure these masks such that a queue is shared between the host and a guest is not only avoidable, it is advisable. It was suggested that this scenario is better handled in user space with management software, but that does not preclude a malicious administrator from using the sysfs interfaces to gain access to a guest's crypto data. It was also suggested that this scenario could be avoided by taking access to the adapter away from the guest and zeroing out the queues prior to the vfio_ap driver releasing the device; however, stealing an adapter in use from a guest as a by-product of an operation is bad and will likely cause problems for the guest unnecessarily. It was decided that the most effective solution with the least number of negative side effects is to prevent the situation at the source.

2. Rationale for hot plug/unplug using matrix mdev sysfs interfaces:
----------------------------------------------------------------
Allowing a user to hot plug/unplug AP resources using the matrix mdev sysfs interfaces circumvents the need to terminate the guest in order to modify its AP configuration. Allowing dynamic configuration makes reconfiguring a guest's AP matrix much less disruptive.

3. Rationale for allowing over-provisioning of AP resources:
-----------------------------------------------------------
Allowing assignment of AP resources to a matrix mdev and ultimately to a guest better models the AP architecture. The architecture does not preclude assignment of unavailable AP resources. If a queue subsequently becomes available while a guest using the matrix mdev to which its APQN is assigned, the guest will be given access to it. If an APQN is dynamically unassigned from the underlying host system, it will automatically become unavailable to the guest.

Change log v17-v18:
------------------
* Added a new document, Documentation/s390/vfio-ap-locking.rst to describe the locking design for this series. This included a patch for adding the new doc to the MAITAINERS file for the VFIO AP maintainers.

* Added Reviewed-by for Halil in patch 6

* Restore filtering in the queue remove callback. (Halil)

* Added patch s390/vfio-ap: hot unplug of AP devices when mdev removed to unplug all AP devices from the guest when the mdev is removed.

* Split patch 9 (s390/vfio-ap: allow hot plug/unplug of AP resources using mdev device) into xx patches:
- allow hot plug/unplug of AP devices when assigned/unassigned

* Replaced v17 patch 08/15, s390/vfio-ap: keep track of active guests with s390/vfio-ap: introduce new mutex to control access to the KVM pointer. No longer tracking guests with a list of struct ap_guest objects. Added a global mutex called kvm_lock which must be held whenever the kvm->lock is taken. (Halil)

* Changed signature of the filtering function to limit the APQNs examined. This change resulted in a cascade of changes to patches 5, 8 and 12. (Halil)

* Removed v17 patch 01/15 (s390/vfio-ap: Set pqap hook when vfio_ap module is loaded). (Halil, Jason).

* Split patch 14 (notify drivers on config changed and scan complete callbacks) into two patches: One to make the AP bus changes and the other to implement the callbacks in the vfio_ap device driver. This was done to facilitate merging the AP bus changes separately from the vfio_ap driver changes. (Harald)

* Removed check of driver flags in the __verify_card_reservations and __verify_queue_reservations functions in ap_bus.c to balance the weighting between the default and vfio_ap drivers. (Harald)

Tony Krowiak (17):
s390/ap: driver callback to indicate resource in use s390/ap: notify drivers on config changed and scan complete callbacks s390/vfio-ap: use new AP bus interface to search for queue devices s390/vfio-ap: move probe and remove callbacks to vfio_ap_ops.c s390/vfio-ap: manage link between queue struct and matrix mdev s390/vfio-ap: introduce shadow APCB
s390/vfio-ap: refresh guest's APCB by filtering APQNs assigned to mdev s390/vfio-ap: allow assignment of unavailable AP queues to mdev device s390/vfio-ap: introduce new mutex to control access to the KVM pointer s390/vfio-ap: allow hot plug/unplug of AP devices when
assigned/unassigned
s390/vfio-ap: hot plug/unplug of AP devices when probed/removed s390/vfio-ap: reset queues after adapter/domain unassignment s390/vfio-ap: implement in-use callback for vfio_ap driver s390/vfio-ap: sysfs attribute to display the guest's matrix s390/vfio-ap: handle config changed and scan complete notification s390/vfio-ap: update docs to include dynamic config support s390/Docs: new doc describing lock usage by the vfio_ap device driver
MAINTAINERS: pick up all vfio_ap docs for VFIO AP maintainers

  Documentation/s390/vfio-ap-locking.rst |  389 +++++++
  Documentation/s390/vfio-ap.rst         |  492 ++++++---
  MAINTAINERS                            |    2 +-
  drivers/s390/crypto/ap_bus.c           |  226 +++-
  drivers/s390/crypto/ap_bus.h           |   16 +
  drivers/s390/crypto/vfio_ap_drv.c      |   71 +-
  drivers/s390/crypto/vfio_ap_ops.c      | 1347 ++++++++++++++++++------
  drivers/s390/crypto/vfio_ap_private.h  |   57 +-
  8 files changed, 2058 insertions(+), 542 deletions(-)

create mode 100644 Documentation/s390/vfio-ap-locking.rst

maintainer
#49 lore raw
Quoted 1 line · Tony Krowiak

PING!! Any takers?

I'm taking a look, starting where I left off on v17, patch #7.

Signature · 1 line
-- Jason J. Herne ([email protected])
maintainer
#8 lore raw

Re: [PATCH v18 07/18] s390/vfio-ap: refresh guest's APCB by filtering APQNs assigned to mdev

In reply to Tony Krowiak
Quoted 43 lines · Tony Krowiak

Refresh the guest's APCB by filtering the APQNs assigned to the matrix mdev that do not reference an AP queue device bound to the vfio_ap device driver. The mdev's APQNs will be filtered according to the following rules:

* The APID of each adapter and the APQI of each domain that is not in the host's AP configuration is filtered out.

* The APID of each adapter comprising an APQN that does not reference a queue device bound to the vfio_ap device driver is filtered. The APQNs are derived from the Cartesian product of the APID of each adapter and APQI of each domain assigned to the mdev.

The control domains that are not assigned to the host's AP configuration will also be filtered before assigning them to the guest's APCB.

---
  drivers/s390/crypto/vfio_ap_ops.c | 96 ++++++++++++++++++++++++++++++-
  1 file changed, 93 insertions(+), 3 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +15 -0
index 4b676a55f203..b67b2f0faeea 100644
@@ -317,6 +317,63 @@ static void vfio_ap_matrix_init(struct ap_config_info *info,
  	matrix->adm_max = info->apxa ? info->Nd : 15;
  }
 
+static void vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev *matrix_mdev)
+{
+	bitmap_and(matrix_mdev->shadow_apcb.adm, matrix_mdev->matrix.adm,
+		   (unsigned long *)matrix_dev->info.adm, AP_DOMAINS);
+}
+
+/*
+ * vfio_ap_mdev_filter_matrix - copy the mdev's AP configuration to the KVM
+ *				guest's APCB then filter the APIDs that do not
+ *				comprise at least one APQN that references a
+ *				queue device bound to the vfio_ap device driver.
+ *
+ * @matrix_mdev: the mdev whose AP configuration is to be filtered.
+ */
+static void vfio_ap_mdev_filter_matrix(unsigned long *apm, unsigned long *aqm,

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_matrix_init(struct ap_config_info *info, · line 344

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +10 -0
+				       struct ap_matrix_mdev *matrix_mdev)
+{
+	int ret;
+	unsigned long apid, apqi, apqn;
+
+	ret = ap_qci(&matrix_dev->info);
+	if (ret)
+		return;
+
+	vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_apcb);

Do you need to call vfio_ap_matrix_init here? It seems to me like this would only be necesarry if apxa could be dynamically added or removed. Here is a copy of vfio_ap_matrix_init, for reference:

static void vfio_ap_matrix_init(struct ap_config_info *info,
struct ap_matrix *matrix)

{
	matrix->apm_max = info->apxa ? info->Na : 63;
	matrix->aqm_max = info->apxa ? info->Nd : 15;
	matrix->adm_max = info->apxa ? info->Nd : 15;
}

It seems like this should be figured out once and stored when the ap_matrix_mdev struct is first created. Unless I'm wrong, and the status of apxa can change dynamically, in which case the maximums would need to be updated somewhere.

Signature · 1 line
-- Jason J. Herne ([email protected])
author maintainer
#9 lore raw

Re: [PATCH v18 07/18] s390/vfio-ap: refresh guest's APCB by filtering APQNs assigned to mdev

Quoted 69 lines · Jason J. Herne
Quoted 46 lines · Tony Krowiak

Refresh the guest's APCB by filtering the APQNs assigned to the matrix mdev that do not reference an AP queue device bound to the vfio_ap device driver. The mdev's APQNs will be filtered according to the following rules:

* The APID of each adapter and the APQI of each domain that is not in the host's AP configuration is filtered out.

* The APID of each adapter comprising an APQN that does not reference a queue device bound to the vfio_ap device driver is filtered. The APQNs are derived from the Cartesian product of the APID of each adapter and APQI of each domain assigned to the mdev.

The control domains that are not assigned to the host's AP configuration will also be filtered before assigning them to the guest's APCB.

---
  drivers/s390/crypto/vfio_ap_ops.c | 96 ++++++++++++++++++++++++++++++-

  1 file changed, 93 insertions(+), 3 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +7 -0
index 4b676a55f203..b67b2f0faeea 100644
@@ -317,6 +317,63 @@ static void vfio_ap_matrix_init(struct 
 ap_config_info *info,
       matrix->adm_max = info->apxa ? info->Nd : 15;
   }
   +static void vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev *matrix_mdev)
+{
+    bitmap_and(matrix_mdev->shadow_apcb.adm, matrix_mdev->matrix.adm,
+           (unsigned long *)matrix_dev->info.adm, AP_DOMAINS);
+}
+
+/*
+ * vfio_ap_mdev_filter_matrix - copy the mdev's AP configuration to 

the KVM
+ *                guest's APCB then filter the APIDs that do not
+ *                comprise at least one APQN that references a
+ *                queue device bound to the vfio_ap device driver.
+ *
+ * @matrix_mdev: the mdev whose AP configuration is to be filtered.
+ */
+static void vfio_ap_mdev_filter_matrix(unsigned long *apm, unsigned long *aqm,
+                       struct ap_matrix_mdev *matrix_mdev)

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_matrix_init(struct ap_config_info *info, · line 344

Quoted 9 lines
drivers/s390/crypto/vfio_ap_ops.c +9 -0
+{
+    int ret;
+    unsigned long apid, apqi, apqn;
+
+    ret = ap_qci(&matrix_dev->info);
+    if (ret)
+        return;
+
+    vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_apcb);

Do you need to call vfio_ap_matrix_init here? It seems to me like this would only be necesarry if apxa could be dynamically added or removed. Here is a copy of vfio_ap_matrix_init, for reference:

static void vfio_ap_matrix_init(struct ap_config_info *info, struct ap_matrix *matrix)
{
    matrix->apm_max = info->apxa ? info->Na : 63; matrix->aqm_max = info->apxa ? info->Nd : 15; matrix->adm_max = info->apxa ? info->Nd : 15; }

Quoted 4 lines

It seems like this should be figured out once and stored when the ap_matrix_mdev struct is first created. Unless I'm wrong, and the status of apxa can change dynamically, in which case the maximums would need to be updated somewhere.

It's an interesting question to which I don't have a definitive answer. I'll run it by our architects. On the other hand, making this call here is not entirely unreasonable and merely superfluous at worst, but I'll look into it.

Tony K

Quoted 2 lines
maintainer
#11 lore raw

Re: [PATCH v18 08/18] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device

In reply to Tony Krowiak
Quoted 26 lines · Tony Krowiak

/**
- * vfio_ap_mdev_verify_no_sharing - verifies that the AP matrix is not configured
+ * vfio_ap_mdev_verify_no_sharing - verify APQNs are not shared by matrix mdevs *
- * @matrix_mdev: the mediated matrix device
+ * @mdev_apm: mask indicating the APIDs of the APQNs to be verified
+ * @mdev_aqm: mask indicating the APQIs of the APQNs to be verified *
- * Verifies that the APQNs derived from the cross product of the AP adapter IDs
- * and AP queue indexes comprising the AP matrix are not configured for another
+ * Verifies that each APQN derived from the Cartesian product of a bitmap of
+ * AP adapter IDs and AP queue indexes is not configured for any matrix
* mediated device. AP queue sharing is not allowed.
*
- * Return: 0 if the APQNs are not shared; otherwise returns -EADDRINUSE.
+ * Return: 0 if the APQNs are not shared; otherwise return -EADDRINUSE. */

drivers/s390/crypto/vfio_ap_ops.c +3 -2
-static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
+static int vfio_ap_mdev_verify_no_sharing(unsigned long *mdev_apm,
+					  unsigned long *mdev_aqm)
  {
-	struct ap_matrix_mdev *lstdev;
+	struct ap_matrix_mdev *matrix_mdev;
  	DECLARE_BITMAP(apm, AP_DEVICES);
  	DECLARE_BITMAP(aqm, AP_DOMAINS);
 

drivers/s390/crypto/vfio_ap_ops.c static struct attribute_group *vfio_ap_mdev_type_groups[] = { · line 570

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +7 -2
-	list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) {
-		if (matrix_mdev == lstdev)
+	list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) {
+		/*
+		 * If the input apm and aqm belong to the matrix_mdev's matrix,
+		 * then move on to the next.
+		 */
+		if (mdev_apm == matrix_mdev->matrix.apm &&
+		    mdev_aqm == matrix_mdev->matrix.aqm)
  			continue;

We may have a problem here. This check seems like it exists to stop you from comparing an mdev's apm/aqm with itself. Obviously comparing an mdev's newly updated apm/aqm with itself would cause a false positive sharing check, right? If this is the case, I think the comment should be changed to reflect that.

Aside from the comment, what stops this particular series of if statements from allowing us to configure a second mdev with the exact same apm/aqm values as an existing mdev? If we do, then this check's continue will short circuit the rest of the function thereby allowing that 2nd mdev even though it should be a sharing violation.

Signature · 1 line
-- Jason J. Herne ([email protected])
author maintainer
#12 lore raw

Re: [PATCH v18 08/18] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device

Quoted 40 lines · Jason J. Herne
Quoted 39 lines · Tony Krowiak

  /**
- * vfio_ap_mdev_verify_no_sharing - verifies that the AP matrix is not configured
+ * vfio_ap_mdev_verify_no_sharing - verify APQNs are not shared by matrix mdevs
   *
- * @matrix_mdev: the mediated matrix device
+ * @mdev_apm: mask indicating the APIDs of the APQNs to be verified
+ * @mdev_aqm: mask indicating the APQIs of the APQNs to be verified *
- * Verifies that the APQNs derived from the cross product of the AP adapter IDs
- * and AP queue indexes comprising the AP matrix are not configured for another
+ * Verifies that each APQN derived from the Cartesian product of a bitmap of
+ * AP adapter IDs and AP queue indexes is not configured for any matrix
   * mediated device. AP queue sharing is not allowed.
   *
- * Return: 0 if the APQNs are not shared; otherwise returns -EADDRINUSE.
+ * Return: 0 if the APQNs are not shared; otherwise return -EADDRINUSE. */
-static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
+static int vfio_ap_mdev_verify_no_sharing(unsigned long *mdev_apm, +                      unsigned long *mdev_aqm)
  {
-    struct ap_matrix_mdev *lstdev;
+    struct ap_matrix_mdev *matrix_mdev;
      DECLARE_BITMAP(apm, AP_DEVICES);
      DECLARE_BITMAP(aqm, AP_DOMAINS);
  -    list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) { -        if (matrix_mdev == lstdev)
+    list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { +        /*
+         * If the input apm and aqm belong to the matrix_mdev's matrix, +         * then move on to the next.
+         */

Quoted 8 lines
Quoted 3 lines

+        if (mdev_apm == matrix_mdev->matrix.apm && +            mdev_aqm == matrix_mdev->matrix.aqm) continue;

We may have a problem here. This check seems like it exists to stop you from comparing an mdev's apm/aqm with itself. Obviously comparing an mdev's newly updated apm/aqm with itself would cause a false positive sharing check, right? If this is the case, I think the comment should be changed to reflect that.

You are correct, this check is performed to prevent comparing an mdev to itself, I'll improve the comment.

Quoted 6 lines

Aside from the comment, what stops this particular series of if statements from allowing us to configure a second mdev with the exact same apm/aqm values as an existing mdev? If we do, then this check's continue will short circuit the rest of the function thereby allowing that 2nd mdev even though it should be a sharing violation.

I don't see how this is possible.

The function above is called from two places: the vfio_ap_mdev_validate_masks() function which is invoked when an adapter or domain is assigned to the mdev; and the vfio_ap_mdev_resource_in_use() function which is a callback registered with the AP bus and is called by the bus when the apmask/aqmask are changed.

In the former case, the addresses passed in are from the apm/aqm fields within the ap_matrix structure. Each ap_matrix structure is a field contained within an ap_matrix_mdev structure, it is not a pointer to storage allocated external to the matrix_mdev, so the apm/aqm addresses passed in from the vfio_ap_mdev_validate_masks() function will be unique to each ap_matrix_mdev structure.

In the latter case, the addresses are passed in by the AP bus and are allocated by the bus and would definitely not be contained within an ap_matrix_mdev since the AP bus doesn't even have access to that structure.

Quoted 2 lines
maintainer
#13 lore raw

Re: [PATCH v18 08/18] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device

Quoted 31 lines · On Mon, 7 Mar 2022 07:31:21 -0500 Tony Krowiak <[email protected]> wrote:
Quoted 30 lines · Jason J. Herne
Quoted 29 lines · Tony Krowiak

  /**
- * vfio_ap_mdev_verify_no_sharing - verifies that the AP matrix is not configured
+ * vfio_ap_mdev_verify_no_sharing - verify APQNs are not shared by matrix mdevs
   *
- * @matrix_mdev: the mediated matrix device
+ * @mdev_apm: mask indicating the APIDs of the APQNs to be verified
+ * @mdev_aqm: mask indicating the APQIs of the APQNs to be verified *
- * Verifies that the APQNs derived from the cross product of the AP adapter IDs
- * and AP queue indexes comprising the AP matrix are not configured for another
+ * Verifies that each APQN derived from the Cartesian product of a bitmap of
+ * AP adapter IDs and AP queue indexes is not configured for any matrix
   * mediated device. AP queue sharing is not allowed.
   *
- * Return: 0 if the APQNs are not shared; otherwise returns -EADDRINUSE.
+ * Return: 0 if the APQNs are not shared; otherwise return -EADDRINUSE. */
-static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev *matrix_mdev)
+static int vfio_ap_mdev_verify_no_sharing(unsigned long *mdev_apm, +                      unsigned long *mdev_aqm)
  {
-    struct ap_matrix_mdev *lstdev;

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+    struct ap_matrix_mdev *matrix_mdev;
      DECLARE_BITMAP(apm, AP_DEVICES);
      DECLARE_BITMAP(aqm, AP_DOMAINS);
  -    list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) { -        if (matrix_mdev == lstdev)
+    list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { +        /*
+         * If the input apm and aqm belong to the matrix_mdev's matrix,

How about:

s/belong to the matrix_mdev's matrix/are fields of the matrix_mdev object/

Quoted 28 lines
Quoted 6 lines
Quoted 5 lines

+         * then move on to the next.
+         */
+        if (mdev_apm == matrix_mdev->matrix.apm && +            mdev_aqm == matrix_mdev->matrix.aqm) continue;

Quoted 8 lines

We may have a problem here. This check seems like it exists to stop you from
comparing an mdev's apm/aqm with itself. Obviously comparing an mdev's newly
updated apm/aqm with itself would cause a false positive sharing check, right?
If this is the case, I think the comment should be changed to reflect that.

You are correct, this check is performed to prevent comparing an mdev to itself, I'll improve the comment.

Quoted 9 lines

Aside from the comment, what stops this particular series of if statements from
allowing us to configure a second mdev with the exact same apm/aqm values as an
existing mdev? If we do, then this check's continue will short circuit the rest
of the function thereby allowing that 2nd mdev even though it should be a sharing violation.

Quoted 1 line

I don't see how this is possible.

I agree with Tony and his explanation.

Furthermore IMHO is relates to the class identity vs equality problem, in a sense that identity always implies equality.

Regards,
Halil

author maintainer
#14 lore raw

Re: [PATCH v18 08/18] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device

Quoted 44 lines · Halil Pasic
Quoted 31 lines · On Mon, 7 Mar 2022 07:31:21 -0500 Tony Krowiak <[email protected]> wrote:
Quoted 30 lines · Jason J. Herne
Quoted 29 lines · Tony Krowiak

  /**
- * vfio_ap_mdev_verify_no_sharing - verifies that the AP matrix is not configured
+ * vfio_ap_mdev_verify_no_sharing - verify APQNs are not shared by matrix mdevs
   *
- * @matrix_mdev: the mediated matrix device
+ * @mdev_apm: mask indicating the APIDs of the APQNs to be verified
+ * @mdev_aqm: mask indicating the APQIs of the APQNs to be verified *
- * Verifies that the APQNs derived from the cross product of the AP adapter IDs
- * and AP queue indexes comprising the AP matrix are not configured for another
+ * Verifies that each APQN derived from the Cartesian product of a bitmap of
+ * AP adapter IDs and AP queue indexes is not configured for any matrix
   * mediated device. AP queue sharing is not allowed.
   *
- * Return: 0 if the APQNs are not shared; otherwise returns

drivers/s390/crypto/vfio_ap_ops.c +1 -2
-EADDRINUSE.
+ * Return: 0 if the APQNs are not shared; otherwise return -EADDRINUSE.
    */
-static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev

*matrix_mdev)
+static int vfio_ap_mdev_verify_no_sharing(unsigned long *mdev_apm, +                      unsigned long *mdev_aqm)
  {
-    struct ap_matrix_mdev *lstdev;

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+    struct ap_matrix_mdev *matrix_mdev;
      DECLARE_BITMAP(apm, AP_DEVICES);
      DECLARE_BITMAP(aqm, AP_DOMAINS);
  -    list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) {
-        if (matrix_mdev == lstdev)
+    list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { +        /*
+         * If the input apm and aqm belong to the matrix_mdev's matrix,

How about:

Quoted 2 lines

s/belong to the matrix_mdev's matrix/are fields of the matrix_mdev object/

This is the comment I wrote:

        /*
         * Comparing an mdev's newly updated apm/aqm with itself would
         * result in a false positive when verifying whether any APQNs
         * are shared; so, if the input apm and aqm belong to the
         * matrix_mdev's matrix, then move on to the next one. */

However, I'd be happy to change it to whatever either of you want.

Quoted 34 lines
Quoted 26 lines
Quoted 7 lines
Quoted 7 lines

+         * then move on to the next.
+         */
+        if (mdev_apm == matrix_mdev->matrix.apm && +            mdev_aqm == matrix_mdev->matrix.aqm)
              continue;

Quoted 8 lines

We may have a problem here. This check seems like it exists to stop you from
comparing an mdev's apm/aqm with itself. Obviously comparing an mdev's newly
updated apm/aqm with itself would cause a false positive sharing check, right?
If this is the case, I think the comment should be changed to reflect that.

You are correct, this check is performed to prevent comparing an mdev to itself, I'll improve the comment.

Quoted 8 lines

Aside from the comment, what stops this particular series of if statements from
allowing us to configure a second mdev with the exact same apm/aqm values as an
existing mdev? If we do, then this check's continue will short circuit the rest
of the function thereby allowing that 2nd mdev even though it should be a sharing violation.

Quoted 1 line

I don't see how this is possible.

I agree with Tony and his explanation.

Furthermore IMHO is relates to the class identity vs equality problem, in a sense that identity always implies equality.

Regards,
Halil

maintainer
#15 lore raw

Re: [PATCH v18 08/18] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device

Quoted 50 lines · On Mon, 7 Mar 2022 09:10:29 -0500 Tony Krowiak <[email protected]> wrote:
Quoted 44 lines · Halil Pasic
Quoted 31 lines · On Mon, 7 Mar 2022 07:31:21 -0500 Tony Krowiak <[email protected]> wrote:
Quoted 30 lines · Jason J. Herne
Quoted 29 lines · Tony Krowiak

  /**
- * vfio_ap_mdev_verify_no_sharing - verifies that the AP matrix is not configured
+ * vfio_ap_mdev_verify_no_sharing - verify APQNs are not shared by matrix mdevs
   *
- * @matrix_mdev: the mediated matrix device
+ * @mdev_apm: mask indicating the APIDs of the APQNs to be verified
+ * @mdev_aqm: mask indicating the APQIs of the APQNs to be verified *
- * Verifies that the APQNs derived from the cross product of the AP adapter IDs
- * and AP queue indexes comprising the AP matrix are not configured for another
+ * Verifies that each APQN derived from the Cartesian product of a bitmap of
+ * AP adapter IDs and AP queue indexes is not configured for any matrix
   * mediated device. AP queue sharing is not allowed.
   *
- * Return: 0 if the APQNs are not shared; otherwise returns

drivers/s390/crypto/vfio_ap_ops.c +1 -2
-EADDRINUSE.
+ * Return: 0 if the APQNs are not shared; otherwise return -EADDRINUSE.
    */
-static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev

*matrix_mdev)
+static int vfio_ap_mdev_verify_no_sharing(unsigned long *mdev_apm, +                      unsigned long *mdev_aqm)
  {
-    struct ap_matrix_mdev *lstdev;

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+    struct ap_matrix_mdev *matrix_mdev;
      DECLARE_BITMAP(apm, AP_DEVICES);
      DECLARE_BITMAP(aqm, AP_DOMAINS);
  -    list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) {
-        if (matrix_mdev == lstdev)
+    list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { +        /*
+         * If the input apm and aqm belong to the matrix_mdev's matrix,

How about:

Quoted 2 lines

s/belong to the matrix_mdev's matrix/are fields of the matrix_mdev object/

This is the comment I wrote:

Quoted 8 lines

        /*
         * Comparing an mdev's newly updated apm/aqm with itself would
         * result in a false positive when verifying whether any APQNs
         * are shared; so, if the input apm and aqm belong to the
         * matrix_mdev's matrix, then move on to the next one. */

However, I'd be happy to change it to whatever either of you want.

What ain't obvious for the comment is that "belong to" actually means composition and not association. In other words, there there is no pointer/indirection involved, a pointer that would tell us what matrix does belong to what matrix_mdev, but rather the matrix is just a part of the matrix_mdev object.

I don't like 'false positive' either, and whether the apm/aqm is newly updated or not is also redundant and confusing in my opinion. When we check because of inuse there is not updated whatever. IMHO the old message was better than this one.

Just my opinion, if you two agree, that this is the way to go, I'm fine with that.

Regards,
Halil

author maintainer
#16 lore raw

Re: [PATCH v18 08/18] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device

Quoted 71 lines · Halil Pasic
Quoted 49 lines · On Mon, 7 Mar 2022 09:10:29 -0500 Tony Krowiak <[email protected]> wrote:
Quoted 44 lines · Halil Pasic
Quoted 31 lines · On Mon, 7 Mar 2022 07:31:21 -0500 Tony Krowiak <[email protected]> wrote:
Quoted 30 lines · Jason J. Herne
Quoted 29 lines · Tony Krowiak

  /**
- * vfio_ap_mdev_verify_no_sharing - verifies that the AP matrix is not configured
+ * vfio_ap_mdev_verify_no_sharing - verify APQNs are not shared by matrix mdevs
   *
- * @matrix_mdev: the mediated matrix device
+ * @mdev_apm: mask indicating the APIDs of the APQNs to be verified
+ * @mdev_aqm: mask indicating the APQIs of the APQNs to be verified *
- * Verifies that the APQNs derived from the cross product of the AP adapter IDs
- * and AP queue indexes comprising the AP matrix are not configured for another
+ * Verifies that each APQN derived from the Cartesian product of a bitmap of
+ * AP adapter IDs and AP queue indexes is not configured for any matrix
   * mediated device. AP queue sharing is not allowed.
   *
- * Return: 0 if the APQNs are not shared; otherwise returns

drivers/s390/crypto/vfio_ap_ops.c +1 -2
-EADDRINUSE.
+ * Return: 0 if the APQNs are not shared; otherwise return -EADDRINUSE.
     */
-static int vfio_ap_mdev_verify_no_sharing(struct ap_matrix_mdev

*matrix_mdev)
+static int vfio_ap_mdev_verify_no_sharing(unsigned long *mdev_apm, +                      unsigned long *mdev_aqm)
  {
-    struct ap_matrix_mdev *lstdev;

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

+    struct ap_matrix_mdev *matrix_mdev;
      DECLARE_BITMAP(apm, AP_DEVICES);
      DECLARE_BITMAP(aqm, AP_DOMAINS);
  -    list_for_each_entry(lstdev, &matrix_dev->mdev_list, node) {
-        if (matrix_mdev == lstdev)
+    list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { +        /*
+         * If the input apm and aqm belong to the matrix_mdev's matrix,

How about:

Quoted 2 lines

s/belong to the matrix_mdev's matrix/are fields of the matrix_mdev object/

This is the comment I wrote:

Quoted 8 lines

        /*
         * Comparing an mdev's newly updated apm/aqm with itself would
         * result in a false positive when verifying whether any APQNs
         * are shared; so, if the input apm and aqm belong to the
         * matrix_mdev's matrix, then move on to the next one. */

However, I'd be happy to change it to whatever either of you want.

What ain't obvious for the comment is that "belong to" actually means composition and not association. In other words, there there is no pointer/indirection involved, a pointer that would tell us what matrix does belong to what matrix_mdev, but rather the matrix is just a part of the matrix_mdev object.

I don't like 'false positive' either, and whether the apm/aqm is newly updated or not is also redundant and confusing in my opinion. When we check because of inuse there is not updated whatever. IMHO the old message was better than this one.

Quoted 5 lines

Just my opinion, if you two agree, that this is the way to go, I'm fine with that.

Regards,
Halil

Feel free to recommend the verbiage for this comment. I'm not married to my comments and am open to anything that helps others to understand what is going on here. It seems obvious to me, but I wrote the code. Obviously, it is not so obvious based on Jason's comments, so maybe someone else can compose a better comment.

Quoted 2 lines
maintainer
#17 lore raw

Re: [PATCH v18 08/18] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device

On Mon, 7 Mar 2022 18:45:45 -0500
Tony Krowiak <[email protected]> wrote:

[..]

Quoted 26 lines
Quoted 26 lines
Quoted 4 lines
Quoted 2 lines

s/belong to the matrix_mdev's matrix/are fields of the matrix_mdev object/

This is the comment I wrote:

Quoted 8 lines

        /*
         * Comparing an mdev's newly updated apm/aqm with itself would
         * result in a false positive when verifying whether any APQNs
         * are shared; so, if the input apm and aqm belong to the
         * matrix_mdev's matrix, then move on to the next one. */

However, I'd be happy to change it to whatever either of you want.

What ain't obvious for the comment is that "belong to" actually means composition and not association. In other words, there there is no pointer/indirection involved, a pointer that would tell us what matrix does belong to what matrix_mdev, but rather the matrix is just a part of the matrix_mdev object.

I don't like 'false positive' either, and whether the apm/aqm is newly updated or not is also redundant and confusing in my opinion. When we check because of inuse there is not updated whatever. IMHO the old message was better than this one.

Just my opinion, if you two agree, that this is the way to go, I'm fine with that.

Quoted 8 lines
Quoted 2 lines

Regards,
Halil

Feel free to recommend the verbiage for this comment. I'm not married to my comments and am open to anything that helps others to understand what is going on here. It seems obvious to me, but I wrote the code. Obviously, it is not so obvious based on Jason's comments, so maybe someone else can compose a better comment.

/*
* If the input apm and aqm are fields of the matrix_mdev object,
* then move on to the next matrix_mdev.
*/

Regards,
Halil

author maintainer
#18 lore raw

Re: [PATCH v18 08/18] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device

Quoted 37 lines · Halil Pasic

On Mon, 7 Mar 2022 18:45:45 -0500
Tony Krowiak <[email protected]> wrote:

[..]

Quoted 26 lines
Quoted 26 lines
Quoted 4 lines
Quoted 2 lines

s/belong to the matrix_mdev's matrix/are fields of the matrix_mdev object/

This is the comment I wrote:

Quoted 8 lines

        /*
         * Comparing an mdev's newly updated apm/aqm with itself would
         * result in a false positive when verifying whether any APQNs
         * are shared; so, if the input apm and aqm belong to the
         * matrix_mdev's matrix, then move on to the next one. */

However, I'd be happy to change it to whatever either of you want.

What ain't obvious for the comment is that "belong to" actually means composition and not association. In other words, there there is no pointer/indirection involved, a pointer that would tell us what matrix does belong to what matrix_mdev, but rather the matrix is just a part of the matrix_mdev object.

I don't like 'false positive' either, and whether the apm/aqm is newly updated or not is also redundant and confusing in my opinion. When we check because of inuse there is not updated whatever. IMHO the old message was better than this one.

Just my opinion, if you two agree, that this is the way to go, I'm fine with that.

Quoted 7 lines
Quoted 2 lines

Regards,
Halil

Feel free to recommend the verbiage for this comment. I'm not married to my comments and am open to anything that helps others to understand what is going on here. It seems obvious to me, but I wrote the code. Obviously, it is not so obvious based on Jason's comments, so maybe someone else can compose a better comment.

Quoted 4 lines

/*
* If the input apm and aqm are fields of the matrix_mdev object,
* then move on to the next matrix_mdev.
*/

Perfect, you write better English than me!

Quoted 3 lines

Regards,
Halil

maintainer
#19 lore raw

Re: [PATCH v18 08/18] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device

In reply to Tony Krowiak
Quoted 16 lines · Tony Krowiak
Quoted 11 lines
Quoted 6 lines

+         * If the input apm and aqm belong to the matrix_mdev's matrix, +         * then move on to the next.
+         */
+        if (mdev_apm == matrix_mdev->matrix.apm &&
+            mdev_aqm == matrix_mdev->matrix.aqm)
              continue;

We may have a problem here. This check seems like it exists to stop you from comparing an mdev's apm/aqm with itself. Obviously comparing an mdev's newly updated apm/aqm with itself would cause a false positive sharing check, right? If this is the case, I think the comment should be changed to reflect that.

You are correct, this check is performed to prevent comparing an mdev to itself, I'll improve the comment.

Quoted 1 line
Quoted 7 lines
Quoted 5 lines

Aside from the comment, what stops this particular series of if statements from allowing us to configure a second mdev with the exact same apm/aqm values as an existing mdev? If we do, then this check's continue will short circuit the rest of the function thereby allowing that 2nd mdev even though it should be a sharing violation.

I don't see how this is possible.

You are correct. I missed the fact that you are comparing pointers here, and not values. Apologies. Now that I understand the code, I agree that this is fine as is.

Signature · 1 line
-- Jason J. Herne ([email protected])
maintainer
#20 lore raw

Re: [PATCH v18 08/18] s390/vfio-ap: allow assignment of unavailable AP queues to mdev device

Quoted 23 lines · On Tue, 8 Mar 2022 10:39:09 -0500 "Jason J. Herne" <[email protected]>…
Quoted 22 lines · Tony Krowiak
Quoted 11 lines
Quoted 6 lines

+         * If the input apm and aqm belong to the matrix_mdev's matrix, +         * then move on to the next.
+         */
+        if (mdev_apm == matrix_mdev->matrix.apm &&
+            mdev_aqm == matrix_mdev->matrix.aqm)
              continue;

We may have a problem here. This check seems like it exists to stop you from comparing an mdev's apm/aqm with itself. Obviously comparing an mdev's newly updated apm/aqm with itself would cause a false positive sharing check, right? If this is the case, I think the comment should be changed to reflect that.

You are correct, this check is performed to prevent comparing an mdev to itself, I'll improve the comment.

Quoted 6 lines

Aside from the comment, what stops this particular series of if statements from allowing us to configure a second mdev with the exact same apm/aqm values as an existing mdev? If we do, then this check's continue will short circuit the rest of the function thereby allowing that 2nd mdev even though it should be a sharing violation.

Quoted 5 lines
Quoted 1 line

I don't see how this is possible.

You are correct. I missed the fact that you are comparing pointers here, and not values. Apologies. Now that I understand the code, I agree that this is fine as is.

I believe clarifying the 'belongs to' vs 'is a part of' stuff is still worthwhile, because 'belongs to' does beg the question you asked. Thus IMHO it is good that you raised the question.

Regards,
Halil

maintainer
#23 lore raw

Re: [PATCH v18 10/18] s390/vfio-ap: allow hot plug/unplug of AP devices when assigned/unassigned

In reply to Tony Krowiak
Quoted 37 lines · Tony Krowiak

Let's allow adapters, domains and control domains to be hot plugged into and hot unplugged from a KVM guest using a matrix mdev when an adapter, domain or control domain is assigned to or unassigned from the matrix mdev.

Whenever an assignment or unassignment of an adapter, domain or control domain is performed, the AP configuration assigned to the matrix mediated device will be filtered and assigned to the AP control block (APCB) that supplies the AP configuration to the guest so that no adapter, domain or control domain that is not in the host's AP configuration nor any APQN that does not reference a queue device bound to the vfio_ap device driver is assigned.

After updating the APCB, if the mdev is in use by a KVM guest, it is hot plugged into the guest to dynamically provide access to the adapters, domains and control domains provided via the newly refreshed APCB.

Keep in mind that the matrix_dev->guests_lock must be taken outside of the matrix_mdev->kvm->lock which in turn must be taken outside of the matrix_dev->mdevs_lock in order to avoid circular lock dependencies (i.e., a lockdep splat).Consequently, the locking order for hot plugging the guest's APCB must be:

matrix_dev->guests_lock => matrix_mdev->kvm->lock => matrix_dev->mdevs_lock

---
  drivers/s390/crypto/vfio_ap_ops.c | 198 +++++++++++++++++++-----------
  1 file changed, 125 insertions(+), 73 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +0 -0
index 623a4b38676d..4c382cd3afc7 100644
@@ -317,10 +317,25 @@ static void vfio_ap_matrix_init(struct ap_config_info *info,
  	matrix->adm_max = info->apxa ? info->Nd : 15;
  }

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_matrix_init(struct ap_config_info *info, · line 327

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +7 -1
 
-static void vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev *matrix_mdev)
+static void vfio_ap_mdev_hotplug_apcb(struct ap_matrix_mdev *matrix_mdev)
  {
+	if (matrix_mdev->kvm)
+		kvm_arch_crypto_set_masks(matrix_mdev->kvm,
+					  matrix_mdev->shadow_apcb.apm,
+					  matrix_mdev->shadow_apcb.aqm,
+					  matrix_mdev->shadow_apcb.adm);
+}

This function updates a kvm guest's apcb. So let's rename it to vfio_ap_update_apcb(). You can also call this function in vfio_ap_mdev_set_kvm, instead of duplicating the code to call kvm_arch_crypto_set_masks().

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_matrix_init(struct ap_config_info *info, · line 339

Quoted 11 lines
drivers/s390/crypto/vfio_ap_ops.c +8 -0
+static bool vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev *matrix_mdev)
+{
+	DECLARE_BITMAP(shadow_adm, AP_DOMAINS);
+
+	bitmap_copy(shadow_adm, matrix_mdev->shadow_apcb.adm, AP_DOMAINS);
  	bitmap_and(matrix_mdev->shadow_apcb.adm, matrix_mdev->matrix.adm,
  		   (unsigned long *)matrix_dev->info.adm, AP_DOMAINS);
+
+	return !bitmap_equal(shadow_adm, matrix_mdev->shadow_apcb.adm,
+			     AP_DOMAINS);
  }

your variable, shadow_adm, should be named original_adm. Since it represents the original value before filtering. This makes the intent much more clear. Same goes for the vars in vfio_ap_mdev_filter_matrix().

...

Quoted 11 lines
drivers/s390/crypto/vfio_ap_ops.c +11 -0
+/**
+ * vfio_ap_mdev_get_locks - acquire the locks required to assign/unassign AP
+ *			    adapters, domains and control domains for an mdev in
+ *			    the proper locking order.
+ *
+ * @matrix_mdev: the matrix mediated device object
+ */
+static void vfio_ap_mdev_get_locks(struct ap_matrix_mdev *matrix_mdev)
+{
+	/* Lock the mutex required to access the KVM guest's state */
+	mutex_lock(&matrix_dev->guests_lock);

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_mdev_link_adapter(struct ap_matrix_mdev *matrix_mdev, · line 664

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +10 -0
+
+	/* If a KVM guest is running, lock the mutex required to plug/unplug the
+	 * AP devices passed through to the guest
+	 */
+	if (matrix_mdev->kvm)
+		mutex_lock(&matrix_mdev->kvm->lock);
+
+	/* The lock required to access the mdev's state */
+	mutex_lock(&matrix_dev->mdevs_lock);
+}

Simplifying the cdoe, and removing duplication by moving the locking code to a function is probably a good thing. But I don't feel like this belongs to this particular patch. In general, a patch should only do one thing, and ideally that one thing should be as small as reasonably possible. This makes the patch easier to read and to review.

I feel like, as much as possible, you should refactor the locking in a series of patches that are all kept together. Ideally, they would be a patch series completely separate from dynamic ap. After all, this series is already at 18 patches. :)

...

Quoted 23 lines

/**
* assign_adapter_store - parses the APID from @buf and sets the
* corresponding bit in the mediated matrix device's APM

drivers/s390/crypto/vfio_ap_ops.c +1 -9
@@ -649,17 +723,9 @@ static ssize_t assign_adapter_store(struct device *dev,
  	int ret;
  	unsigned long apid;
  	DECLARE_BITMAP(apm, AP_DEVICES);
-
  	struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev);
 
-	mutex_lock(&matrix_dev->guests_lock);
-	mutex_lock(&matrix_dev->mdevs_lock);
-
-	/* If the KVM guest is running, disallow assignment of adapter */
-	if (matrix_mdev->kvm) {
-		ret = -EBUSY;
-		goto done;
-	}
+	vfio_ap_mdev_get_locks(matrix_mdev);
 
  	ret = kstrtoul(buf, 0, &apid);
  	if (ret)

drivers/s390/crypto/vfio_ap_ops.c static ssize_t assign_adapter_store(struct device *dev, · line 741

Quoted 7 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -2
@@ -671,8 +737,6 @@ static ssize_t assign_adapter_store(struct device *dev,
  	}
 
  	set_bit_inv(apid, matrix_mdev->matrix.apm);
-	memset(apm, 0, sizeof(apm));
-	set_bit_inv(apid, apm);
 
  	ret = vfio_ap_mdev_validate_masks(matrix_mdev);

It looks like you moved the memset() and set_bit_inv() to be closer to where "apm" is used, namely, the call to vfio_ap_mdev_filter_matrix(). Any reason you cannot move it down under the call to vfio_ap_mdev_link_adapter()? That would get it even closer to where it is used.

Also, I think renaming apm to apm_delta or apm_diff makes sense here. After all, it is the difference between the original apm, and the new apm. The new apm has an extra bit for the newly added adapter. Do I have that right? If so, I think renaming the variable will make the code clearer.

Both of the above comments also apply to assign_domain_store().

Signature · 1 line
-- Jason J. Herne ([email protected])
author maintainer
#24 lore raw

Re: [PATCH v18 10/18] s390/vfio-ap: allow hot plug/unplug of AP devices when assigned/unassigned

Quoted 44 lines · Jason J. Herne
Quoted 43 lines · Tony Krowiak

Let's allow adapters, domains and control domains to be hot plugged into and hot unplugged from a KVM guest using a matrix mdev when an adapter, domain or control domain is assigned to or unassigned from the matrix mdev.

Whenever an assignment or unassignment of an adapter, domain or control domain is performed, the AP configuration assigned to the matrix mediated device will be filtered and assigned to the AP control block (APCB) that supplies the AP configuration to the guest so that no adapter, domain or control domain that is not in the host's AP configuration nor any APQN that does not reference a queue device bound to the vfio_ap device driver is assigned.

After updating the APCB, if the mdev is in use by a KVM guest, it is hot plugged into the guest to dynamically provide access to the adapters, domains and control domains provided via the newly refreshed APCB.

Keep in mind that the matrix_dev->guests_lock must be taken outside of the matrix_mdev->kvm->lock which in turn must be taken outside of the matrix_dev->mdevs_lock in order to avoid circular lock dependencies (i.e., a lockdep splat).Consequently, the locking order for hot plugging the guest's APCB must be:

matrix_dev->guests_lock => matrix_mdev->kvm->lock => matrix_dev->mdevs_lock

---
  drivers/s390/crypto/vfio_ap_ops.c | 198 +++++++++++++++++++-----------

  1 file changed, 125 insertions(+), 73 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +2 -0
index 623a4b38676d..4c382cd3afc7 100644
@@ -317,10 +317,25 @@ static void vfio_ap_matrix_init(struct 
 ap_config_info *info,
       matrix->adm_max = info->apxa ? info->Nd : 15;
   }
   -static void vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev *matrix_mdev)
+static void vfio_ap_mdev_hotplug_apcb(struct ap_matrix_mdev 
 *matrix_mdev)
   {
+    if (matrix_mdev->kvm)
Quoted 8 lines

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_matrix_init(struct ap_config_info *info, · line 327

Quoted 5 lines
drivers/s390/crypto/vfio_ap_ops.c +5 -0
+        kvm_arch_crypto_set_masks(matrix_mdev->kvm,
+                      matrix_mdev->shadow_apcb.apm,
+                      matrix_mdev->shadow_apcb.aqm,
+                      matrix_mdev->shadow_apcb.adm);
+}

This function updates a kvm guest's apcb. So let's rename it to vfio_ap_update_apcb().

The idea was to indicate that the AP adapters, domains and control domains configured in the shadow APCB are being hot plugged into a running guest. Having said that, I can see your point. I'm not married to the function name, but I would prefer to go with 'vfio_ap_update_guest_apcb()' to distinguish between the shadow and the real apcb.

Quoted 2 lines

You can also call this function in vfio_ap_mdev_set_kvm, instead of duplicating the code to call kvm_arch_crypto_set_masks().

The reason I didn't do that is because we've already verified the matrix_mdev->kvm in kvm_arch_crypto_set_masks(). I'm not sure what it buys us, but I'm not adverse to making the change.

Quoted 4 lines
Quoted 16 lines

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_matrix_init(struct ap_config_info *info, · line 339

Quoted 12 lines
drivers/s390/crypto/vfio_ap_ops.c +8 -0
+static bool vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev 
 *matrix_mdev)
+{
+    DECLARE_BITMAP(shadow_adm, AP_DOMAINS);
+
+    bitmap_copy(shadow_adm, matrix_mdev->shadow_apcb.adm, AP_DOMAINS);
       bitmap_and(matrix_mdev->shadow_apcb.adm, matrix_mdev->matrix.adm,
              (unsigned long *)matrix_dev->info.adm, AP_DOMAINS);
+
+    return !bitmap_equal(shadow_adm, matrix_mdev->shadow_apcb.adm,
+                 AP_DOMAINS);
   }

your variable, shadow_adm, should be named original_adm. Since it represents the original value before filtering. This makes the intent much more clear. Same goes for the vars in vfio_ap_mdev_filter_matrix().

Makes sense, but I think I'll go with prev_shadow_apm, prev_shadow_aqm and prev_shadow_adm. That seems more accurate since these are not the original copies of the bitmaps, but copies of the previous versions prior to filtering.

Quoted 32 lines

...

Quoted 19 lines

+/**
+ * vfio_ap_mdev_get_locks - acquire the locks required to assign/unassign AP
+ *                adapters, domains and control domains for an mdev in
+ *                the proper locking order.
+ *
+ * @matrix_mdev: the matrix mediated device object
+ */

drivers/s390/crypto/vfio_ap_ops.c +6 -0
+static void vfio_ap_mdev_get_locks(struct ap_matrix_mdev *matrix_mdev)
+{
+    /* Lock the mutex required to access the KVM guest's state */
+    mutex_lock(&matrix_dev->guests_lock);
+
+    /* If a KVM guest is running, lock the mutex required to 

plug/unplug the
+     * AP devices passed through to the guest
+     */
+    if (matrix_mdev->kvm)
+        mutex_lock(&matrix_mdev->kvm->lock);

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_mdev_link_adapter(struct ap_matrix_mdev *matrix_mdev, · line 664

Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +4 -0
+
+    /* The lock required to access the mdev's state */
+    mutex_lock(&matrix_dev->mdevs_lock);
+}

Simplifying the cdoe, and removing duplication by moving the locking code to a function is probably a good thing. But I don't feel like this belongs to this particular patch. In general, a patch should only do one thing, and ideally that one thing should be as small as reasonably possible. This makes the patch easier to read and to review.

Quoted 4 lines

I feel like, as much as possible, you should refactor the locking in a series of patches that are all kept together. Ideally, they would be a patch series completely separate from dynamic ap. After all, this series is already at 18 patches. :)

I'm going to have to disagree, this locking scheme makes no sense outside of this series. It is only necessary because we now update a guest's APCB whenever an adapter, domain or control domain is assigned or unassigned, when a queue device is probed or removed and when the vfio_ap driver is notified that the host's AP configuration has changed.

Prior to this series, a guest's APCB was updated only when the vfio_ap driver was notified that the KVM pointer was set or cleared, so it was
only necessary to ensure the kvm->lock is taken before the matrix_dev->lock in the functions that handle the VFIO_GROUP_NOTIFY_SET_KVM group notification event. Prior to this, a patch series to introduce the matrix_dev->guests lock would make no sense because it is not needed to enforce the locking order in those functions listed in the previous paragraph because we didn't update the guest's APCB in those functions.

Quoted 27 lines

...

Quoted 25 lines

  /**
   * assign_adapter_store - parses the APID from @buf and sets the
   * corresponding bit in the mediated matrix device's APM

drivers/s390/crypto/vfio_ap_ops.c +1 -8
@@ -649,17 +723,9 @@ static ssize_t assign_adapter_store(struct 
 device *dev,
       int ret;
       unsigned long apid;
       DECLARE_BITMAP(apm, AP_DEVICES);
-
       struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev);
   -    mutex_lock(&matrix_dev->guests_lock);
-    mutex_lock(&matrix_dev->mdevs_lock);
-
-    /* If the KVM guest is running, disallow assignment of adapter */
-    if (matrix_mdev->kvm) {
-        ret = -EBUSY;
-        goto done;
-    }
+    vfio_ap_mdev_get_locks(matrix_mdev);
         ret = kstrtoul(buf, 0, &apid);
       if (ret)
@@ -671,8 +737,6 @@ static ssize_t assign_adapter_store(struct device 
 *dev,
       }
         set_bit_inv(apid, matrix_mdev->matrix.apm);
Quoted 8 lines

drivers/s390/crypto/vfio_ap_ops.c static ssize_t assign_adapter_store(struct device *dev, · line 741

Quoted 3 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -2
-    memset(apm, 0, sizeof(apm));
-    set_bit_inv(apid, apm);
         ret = vfio_ap_mdev_validate_masks(matrix_mdev);

It looks like you moved the memset() and set_bit_inv() to be closer to where "apm" is used, namely, the call to vfio_ap_mdev_filter_matrix(). Any reason you cannot move it down under the call to vfio_ap_mdev_link_adapter()? That would get it even closer to where it is used.

I didn't move it to be closer to where it is used, I moved it because it was not necessary to do the memset/set_bit_inv when not necessary to do so. Having said that, it can definitely be moved after the vfio_ap_mdev_link_adapter().

Quoted 5 lines

Also, I think renaming apm to apm_delta or apm_diff makes sense here. After all, it is the difference between the original apm, and the new apm. The new apm has an extra bit for the newly added adapter. Do I have that right? If so, I think renaming the variable will make the code clearer.

The purpose of this bitmap is to limit the filtering to the new APID being assigned because there is no need to do filtering of adapters already assigned; so, it is not really a new apm per se. It might be more accurate to call it new_apid or new_apids, although there will only be one bit set in the bitmap.

Quoted 3 lines

Both of the above comments also apply to assign_domain_store().

Ditto on the responses.

maintainer
#25 lore raw

Re: [PATCH v18 10/18] s390/vfio-ap: allow hot plug/unplug of AP devices when assigned/unassigned

Quoted 61 lines · Tony Krowiak
Quoted 42 lines · Jason J. Herne
Quoted 41 lines · Tony Krowiak

Let's allow adapters, domains and control domains to be hot plugged into and hot unplugged from a KVM guest using a matrix mdev when an adapter, domain or control domain is assigned to or unassigned from the matrix mdev.

Whenever an assignment or unassignment of an adapter, domain or control domain is performed, the AP configuration assigned to the matrix mediated device will be filtered and assigned to the AP control block (APCB) that supplies the AP configuration to the guest so that no adapter, domain or control domain that is not in the host's AP configuration nor any APQN that does not reference a queue device bound to the vfio_ap device driver is assigned.

After updating the APCB, if the mdev is in use by a KVM guest, it is hot plugged into the guest to dynamically provide access to the adapters, domains and control domains provided via the newly refreshed APCB.

Keep in mind that the matrix_dev->guests_lock must be taken outside of the matrix_mdev->kvm->lock which in turn must be taken outside of the matrix_dev->mdevs_lock in order to avoid circular lock dependencies (i.e., a lockdep splat).Consequently, the locking order for hot plugging the guest's APCB must be:

matrix_dev->guests_lock => matrix_mdev->kvm->lock => matrix_dev->mdevs_lock

---
  drivers/s390/crypto/vfio_ap_ops.c | 198 +++++++++++++++++++-----------

  1 file changed, 125 insertions(+), 73 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +2 -0
index 623a4b38676d..4c382cd3afc7 100644
@@ -317,10 +317,25 @@ static void vfio_ap_matrix_init(struct ap_config_info *info,
       matrix->adm_max = info->apxa ? info->Nd : 15;
   }
   -static void vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev *matrix_mdev)
+static void vfio_ap_mdev_hotplug_apcb(struct ap_matrix_mdev *matrix_mdev)
   {
+    if (matrix_mdev->kvm)
Quoted 8 lines

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_matrix_init(struct ap_config_info *info, · line 327

Quoted 5 lines
drivers/s390/crypto/vfio_ap_ops.c +5 -0
+        kvm_arch_crypto_set_masks(matrix_mdev->kvm,
+                      matrix_mdev->shadow_apcb.apm,
+                      matrix_mdev->shadow_apcb.aqm,
+                      matrix_mdev->shadow_apcb.adm);
+}

This function updates a kvm guest's apcb. So let's rename it to vfio_ap_update_apcb().

The idea was to indicate that the AP adapters, domains and control domains configured in the shadow APCB are being hot plugged into a running guest. Having said that, I can see your point. I'm not married to the function name, but I would prefer to go with 'vfio_ap_update_guest_apcb()' to distinguish between the shadow and the real apcb.

Quoted 7 lines
Quoted 2 lines

You can also call this function in vfio_ap_mdev_set_kvm, instead of duplicating the code to call kvm_arch_crypto_set_masks().

The reason I didn't do that is because we've already verified the matrix_mdev->kvm in kvm_arch_crypto_set_masks(). I'm not sure what it buys us, but I'm not adverse to making the change.

It avoids code duplication which makes the driver smaller, and slightly easier to read. It also reduces rework effort if/when mask handling ever changes.

Quoted 20 lines
Quoted 4 lines
Quoted 15 lines

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_matrix_init(struct ap_config_info *info, · line 339

Quoted 11 lines
drivers/s390/crypto/vfio_ap_ops.c +8 -0
+static bool vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev *matrix_mdev)
+{
+    DECLARE_BITMAP(shadow_adm, AP_DOMAINS);
+
+    bitmap_copy(shadow_adm, matrix_mdev->shadow_apcb.adm, AP_DOMAINS);
       bitmap_and(matrix_mdev->shadow_apcb.adm, matrix_mdev->matrix.adm,
              (unsigned long *)matrix_dev->info.adm, AP_DOMAINS);
+
+    return !bitmap_equal(shadow_adm, matrix_mdev->shadow_apcb.adm,
+                 AP_DOMAINS);
   }

your variable, shadow_adm, should be named original_adm. Since it represents the original value before filtering. This makes the intent much more clear. Same goes for the vars in vfio_ap_mdev_filter_matrix().

Quoted 3 lines

Makes sense, but I think I'll go with prev_shadow_apm, prev_shadow_aqm and prev_shadow_adm. That seems more accurate since these are not the original copies of the bitmaps, but copies of the previous versions prior to filtering.

That works for me :) Thanks! In general, I like to avoid generic variable names like "mask" or "thing" whenever possible. Especially if I'm dealing with multiple instances of the same type of data within the same scope. Giving each variable a specific name can really help de-obfuscate the code.

Quoted 41 lines
Quoted 30 lines

...

Quoted 11 lines
drivers/s390/crypto/vfio_ap_ops.c +11 -0
+/**
+ * vfio_ap_mdev_get_locks - acquire the locks required to assign/unassign AP
+ *                adapters, domains and control domains for an mdev in
+ *                the proper locking order.
+ *
+ * @matrix_mdev: the matrix mediated device object
+ */
+static void vfio_ap_mdev_get_locks(struct ap_matrix_mdev *matrix_mdev)
+{
+    /* Lock the mutex required to access the KVM guest's state */
+    mutex_lock(&matrix_dev->guests_lock);

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_mdev_link_adapter(struct ap_matrix_mdev *matrix_mdev, · line 664

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +10 -0
+
+    /* If a KVM guest is running, lock the mutex required to plug/unplug the
+     * AP devices passed through to the guest
+     */
+    if (matrix_mdev->kvm)
+        mutex_lock(&matrix_mdev->kvm->lock);
+
+    /* The lock required to access the mdev's state */
+    mutex_lock(&matrix_dev->mdevs_lock);
+}

Simplifying the cdoe, and removing duplication by moving the locking code to a function is probably a good thing. But I don't feel like this belongs to this particular patch. In general, a patch should only do one thing, and ideally that one thing should be as small as reasonably possible. This makes the patch easier to read and to review.

Quoted 4 lines

I feel like, as much as possible, you should refactor the locking in a series of patches that are all kept together. Ideally, they would be a patch series completely separate from dynamic ap. After all, this series is already at 18 patches. :)

I'm going to have to disagree, this locking scheme makes no sense outside of this series. It is only necessary because we now update a guest's APCB whenever an adapter, domain or control domain is assigned or unassigned, when a queue device is probed or removed and when the vfio_ap driver is notified that the host's AP configuration has changed.

Quoted 9 lines

Prior to this series, a guest's APCB was updated only when the vfio_ap driver was notified that the KVM pointer was set or cleared, so it was
only necessary to ensure the kvm->lock is taken before the matrix_dev->lock in the functions that handle the VFIO_GROUP_NOTIFY_SET_KVM group notification event. Prior to this, a patch series to introduce the matrix_dev->guests lock would make no sense because it is not needed to enforce the locking order in those functions listed in the previous paragraph because we didn't update the guest's APCB in those functions.

I don't understand the lock code enough to argue a whole lot here :) But I do still think, at the very least, that your refactoring of the locking into get_locks/put_locks functions really does belong in a separate patch. Refactoring is not directly related to the hotplug/unplug. Also, this is not a minor refactor. This refactoring touches the code all over the place and really just adds noise to this patch. That noise makes it harder to review.

Quoted 43 lines
Quoted 24 lines

...

Quoted 23 lines

  /**
   * assign_adapter_store - parses the APID from @buf and sets the
   * corresponding bit in the mediated matrix device's APM

drivers/s390/crypto/vfio_ap_ops.c +1 -8
@@ -649,17 +723,9 @@ static ssize_t assign_adapter_store(struct device *dev,
       int ret;
       unsigned long apid;
       DECLARE_BITMAP(apm, AP_DEVICES);
-
       struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev);
   -    mutex_lock(&matrix_dev->guests_lock);
-    mutex_lock(&matrix_dev->mdevs_lock);
-
-    /* If the KVM guest is running, disallow assignment of adapter */
-    if (matrix_mdev->kvm) {
-        ret = -EBUSY;
-        goto done;
-    }
+    vfio_ap_mdev_get_locks(matrix_mdev);
         ret = kstrtoul(buf, 0, &apid);
       if (ret)
@@ -671,8 +737,6 @@ static ssize_t assign_adapter_store(struct device *dev,
       }
         set_bit_inv(apid, matrix_mdev->matrix.apm);
Quoted 8 lines

drivers/s390/crypto/vfio_ap_ops.c static ssize_t assign_adapter_store(struct device *dev, · line 741

Quoted 3 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -2
-    memset(apm, 0, sizeof(apm));
-    set_bit_inv(apid, apm);
         ret = vfio_ap_mdev_validate_masks(matrix_mdev);

It looks like you moved the memset() and set_bit_inv() to be closer to where "apm" is used, namely, the call to vfio_ap_mdev_filter_matrix(). Any reason you cannot move it down under the call to vfio_ap_mdev_link_adapter()? That would get it even closer to where it is used.

I didn't move it to be closer to where it is used, I moved it because it was not necessary to do the memset/set_bit_inv when not necessary to do so. Having said that, it can definitely be moved after the vfio_ap_mdev_link_adapter().

Quoted 5 lines

Also, I think renaming apm to apm_delta or apm_diff makes sense here. After all, it is the difference between the original apm, and the new apm. The new apm has an extra bit for the newly added adapter. Do I have that right? If so, I think renaming the variable will make the code clearer.

Quoted 4 lines

The purpose of this bitmap is to limit the filtering to the new APID being assigned because there is no need to do filtering of adapters already assigned; so, it is not really a new apm per se. It might be more accurate to call it new_apid or new_apids, although there will only be one bit set in the bitmap.

My main concern here was generic variables names. The "new_apm" will have exactly one
new bit set. That bit is the delta (or the difference) between the previously existing apm, and the new apm, which will be the result of adding in whatever the "apid" bit is. Therefore, it really is a delta, right? This was the basis for my suggestion of the name. Its really not the "new" apm... its the difference between the old and new.

Signature · 1 line
-- Jason J. Herne ([email protected])
maintainer
#29 lore raw

Re: [PATCH v18 12/18] s390/vfio-ap: reset queues after adapter/domain unassignment

In reply to Tony Krowiak
Quoted 9 lines · Tony Krowiak

When an adapter or domain is unassigned from an mdev providing the AP configuration to a running KVM guest, one or more of the guest's queues may get dynamically removed. Since the removed queues could get re-assigned to another mdev, they need to be reset. So, when an adapter or domain is unassigned from the mdev, the queues that are removed from the guest's AP configuration will be reset.

---

...

Quoted 26 lines
drivers/s390/crypto/vfio_ap_ops.c +25 -0
+static void vfio_ap_unlink_apqn_fr_mdev(struct ap_matrix_mdev *matrix_mdev,
+					unsigned long apid, unsigned long apqi,
+					struct ap_queue_table *qtable)
+{
+	struct vfio_ap_queue *q;
+
+	q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi));
+	/* If the queue is assigned to the matrix mdev, unlink it. */
+	if (q)
+		vfio_ap_unlink_queue_fr_mdev(q);
+
+	/* If the queue is assigned to the APCB, store it in @qtable. */
+	if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) &&
+	    test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm))
+		hash_add(qtable->queues, &q->mdev_qnode, q->apqn);
+}
+
+/**
+ * vfio_ap_mdev_unlink_adapter - unlink all queues associated with unassigned
+ *				 adapter from the matrix mdev to which the
+ *				 adapter was assigned.
+ * @matrix_mdev: the matrix mediated device to which the adapter was assigned.
+ * @apid: the APID of the unassigned adapter.
+ * @qtable: table for storing queues associated with unassigned adapter.
+ */

drivers/s390/crypto/vfio_ap_ops.c static ssize_t assign_adapter_store(struct device *dev, · line 792

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +6 -1
  static void vfio_ap_mdev_unlink_adapter(struct ap_matrix_mdev *matrix_mdev,
-					unsigned long apid)
+					unsigned long apid,
+					struct ap_queue_table *qtable)
  {
  	unsigned long apqi;
+
+	for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS)
+		vfio_ap_unlink_apqn_fr_mdev(matrix_mdev, apid, apqi, qtable);
+}

Here is an alternate version of the above two functions that stops the profileration of the qtables variable into vfio_ap_unlink_apqn_fr_mdev. It may seem like a change with no benefit, but it simplifies things a bit and avoids the reader from having to sink three functions deep to find out where qtables is used. This is 100% untested.

static bool vfio_ap_unlink_apqn_fr_mdev(struct ap_matrix_mdev *matrix_mdev,
unsigned long apid, unsigned long apqi)

{
	struct vfio_ap_queue *q;

	q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi));
	/* If the queue is assigned to the matrix mdev, unlink it. */
	if (q) {
		vfio_ap_unlink_queue_fr_mdev(q);
		return true;
	}
	return false;
}

static void vfio_ap_mdev_unlink_adapter(struct ap_matrix_mdev *matrix_mdev,
unsigned long apid, struct ap_queue_table *qtable)

{
	unsigned long apqi;
	bool unlinked;

	for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) {
		unlinked = vfio_ap_unlink_apqn_fr_mdev(matrix_mdev, apid, apqi, qtable);

		if (unlinked && qtable) {
			if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) &&
			    test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm))
				hash_add(qtable->queues, &q->mdev_qnode,
					 q->apqn);
		}
	}
}
Quoted 11 lines
drivers/s390/crypto/vfio_ap_ops.c +10 -0
+static void vfio_ap_mdev_hot_unplug_adapter(struct ap_matrix_mdev *matrix_mdev,
+					    unsigned long apid)
+{
+	int bkt;
  	struct vfio_ap_queue *q;
+	struct ap_queue_table qtable;
+	hash_init(qtable.queues);
+	vfio_ap_mdev_unlink_adapter(matrix_mdev, apid, &qtable);
+
+	if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm)) {
+		clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm);

drivers/s390/crypto/vfio_ap_ops.c static ssize_t assign_adapter_store(struct device *dev, · line 811

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +4 -4
+		vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+	}
 
-	for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) {
-		q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi));
+	vfio_ap_mdev_reset_queues(&qtable);
 
-		if (q)
-			vfio_ap_mdev_unlink_queue(q);
+	hash_for_each(qtable.queues, bkt, q, mdev_qnode) {

This comment applies to all instances of btk: What is btk? Can we come up with a more descriptive name?

Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +2 -0
+		vfio_ap_unlink_mdev_fr_queue(q);
+		hash_del(&q->mdev_qnode);
  	}
  }

...

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_mdev_unset_kvm(struct ap_matrix_mdev *matrix_mdev, · line 1326

Quoted 11 lines
drivers/s390/crypto/vfio_ap_ops.c +3 -3
@@ -1273,9 +1320,9 @@ static void vfio_ap_mdev_unset_kvm(struct ap_matrix_mdev *matrix_mdev,
  		mutex_lock(&kvm->lock);
  		mutex_lock(&matrix_dev->mdevs_lock);
 
-		kvm_arch_crypto_clear_masks(kvm);
-		vfio_ap_mdev_reset_queues(matrix_mdev);
-		kvm_put_kvm(kvm);
+		kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
+		vfio_ap_mdev_reset_queues(&matrix_mdev->qtable);
+		kvm_put_kvm(matrix_mdev->kvm);
  		matrix_mdev->kvm = NULL;

I understand changing the call to vfio_ap_mdev_reset_queues, but why are we changing the kvm pointer on the surrounding lines?

drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, unsigned int retry) · line 1378

Quoted 7 lines
drivers/s390/crypto/vfio_ap_ops.c +1 -0
  		mutex_unlock(&matrix_dev->mdevs_lock);
@@ -1328,14 +1375,17 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, unsigned int retry)
 
  	if (!q)
  		return 0;
+	q->reset_rc = 0;

This line seems unnecessary. You set q->reset_rc in every single case below, so this 0 will always get overwritten.

Quoted 34 lines
drivers/s390/crypto/vfio_ap_ops.c +11 -2
  retry_zapq:
  	status = ap_zapq(q->apqn);
  	switch (status.response_code) {
  	case AP_RESPONSE_NORMAL:
  		ret = 0;
+		q->reset_rc = status.response_code;
  		break;
  	case AP_RESPONSE_RESET_IN_PROGRESS:
+		q->reset_rc = status.response_code;
  		if (retry--) {
  			msleep(20);
  			goto retry_zapq;
@@ -1345,13 +1395,20 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, unsigned int retry)
  	case AP_RESPONSE_Q_NOT_AVAIL:
  	case AP_RESPONSE_DECONFIGURED:
  	case AP_RESPONSE_CHECKSTOPPED:
-		WARN_ON_ONCE(status.irq_enabled);
+		WARN_ONCE(status.irq_enabled,
+			  "PQAP/ZAPQ for %02x.%04x failed with rc=%u while IRQ enabled",
+			  AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
+			  status.response_code);
+		q->reset_rc = status.response_code;
  		ret = -EBUSY;
  		goto free_resources;
  	default:
  		/* things are really broken, give up */
-		WARN(true, "PQAP/ZAPQ completed with invalid rc (%x)\n",
+		WARN(true,
+		     "PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n",
+		     AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn),
  		     status.response_code);
+		q->reset_rc = status.response_code;
  		return -EIO;
  	}

...

Signature · 1 line
-- Jason J. Herne ([email protected])
author maintainer
#26 lore raw

Re: [PATCH v18 10/18] s390/vfio-ap: allow hot plug/unplug of AP devices when assigned/unassigned

In reply to Jason J. Herne
Quoted 68 lines · Jason J. Herne
Quoted 67 lines · Tony Krowiak
Quoted 45 lines · Jason J. Herne
Quoted 44 lines · Tony Krowiak

Let's allow adapters, domains and control domains to be hot plugged into and hot unplugged from a KVM guest using a matrix mdev when an adapter, domain or control domain is assigned to or unassigned from the matrix mdev.

Whenever an assignment or unassignment of an adapter, domain or control domain is performed, the AP configuration assigned to the matrix mediated device will be filtered and assigned to the AP control block (APCB) that supplies the AP configuration to the guest so that no adapter, domain or control domain that is not in the host's AP configuration nor any APQN that does not reference a queue device bound to the vfio_ap device driver is assigned.

After updating the APCB, if the mdev is in use by a KVM guest, it is hot plugged into the guest to dynamically provide access to the adapters, domains and control domains provided via the newly refreshed APCB.

Keep in mind that the matrix_dev->guests_lock must be taken outside of the matrix_mdev->kvm->lock which in turn must be taken outside of the matrix_dev->mdevs_lock in order to avoid circular lock dependencies (i.e., a lockdep splat).Consequently, the locking order for hot plugging the guest's APCB must be:

matrix_dev->guests_lock => matrix_mdev->kvm->lock => matrix_dev->mdevs_lock

---
  drivers/s390/crypto/vfio_ap_ops.c | 198
+++++++++++++++++++-----------

  1 file changed, 125 insertions(+), 73 deletions(-)
drivers/s390/crypto/vfio_ap_ops.c +2 -0
index 623a4b38676d..4c382cd3afc7 100644
@@ -317,10 +317,25 @@ static void vfio_ap_matrix_init(struct 
 ap_config_info *info,
       matrix->adm_max = info->apxa ? info->Nd : 15;
   }
   -static void vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev *matrix_mdev)
+static void vfio_ap_mdev_hotplug_apcb(struct ap_matrix_mdev 
 *matrix_mdev)
   {
+    if (matrix_mdev->kvm)
Quoted 8 lines

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_matrix_init(struct ap_config_info *info, · line 327

Quoted 5 lines
drivers/s390/crypto/vfio_ap_ops.c +5 -0
+        kvm_arch_crypto_set_masks(matrix_mdev->kvm,
+                      matrix_mdev->shadow_apcb.apm,
+                      matrix_mdev->shadow_apcb.aqm,
+                      matrix_mdev->shadow_apcb.adm);
+}

This function updates a kvm guest's apcb. So let's rename it to vfio_ap_update_apcb().

The idea was to indicate that the AP adapters, domains and control domains configured in the shadow APCB are being hot plugged into a running guest. Having said that, I can see your point. I'm not married to the function name, but I would prefer to go with 'vfio_ap_update_guest_apcb()' to distinguish between the shadow and the real apcb.

Quoted 2 lines

You can also call this function in vfio_ap_mdev_set_kvm, instead of duplicating the code to call kvm_arch_crypto_set_masks().

Quoted 8 lines
Quoted 4 lines

The reason I didn't do that is because we've already verified the matrix_mdev->kvm in kvm_arch_crypto_set_masks(). I'm not sure what it buys us, but I'm not adverse to making the change.

It avoids code duplication which makes the driver smaller, and slightly easier to read. It also reduces rework effort if/when mask handling ever changes.

Good points.

Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 17 lines
drivers/s390/crypto/vfio_ap_ops.c +8 -0
+static bool vfio_ap_mdev_filter_cdoms(struct ap_matrix_mdev 
 *matrix_mdev)
+{
+    DECLARE_BITMAP(shadow_adm, AP_DOMAINS);
+
+    bitmap_copy(shadow_adm, matrix_mdev->shadow_apcb.adm, 
 AP_DOMAINS);
       bitmap_and(matrix_mdev->shadow_apcb.adm, matrix_mdev->matrix.adm,
              (unsigned long *)matrix_dev->info.adm, AP_DOMAINS);
+
+    return !bitmap_equal(shadow_adm, matrix_mdev->shadow_apcb.adm,
+                 AP_DOMAINS);
Quoted 1 line

  }

your variable, shadow_adm, should be named original_adm. Since it represents the original value before filtering. This makes the intent much more clear. Same goes for the vars in vfio_ap_mdev_filter_matrix().

Quoted 8 lines
Quoted 3 lines

Makes sense, but I think I'll go with prev_shadow_apm, prev_shadow_aqm and prev_shadow_adm. That seems more accurate since these are not the original copies of the bitmaps, but copies of the previous versions prior to filtering.

That works for me :) Thanks! In general, I like to avoid generic variable names like "mask" or "thing" whenever possible. Especially if I'm dealing with multiple instances of the same type of data within the same scope. Giving each variable a specific name can really help de-obfuscate the code.

Agreed. It has been done.

Quoted 56 lines
Quoted 46 lines
Quoted 35 lines

...

Quoted 21 lines

+/**
+ * vfio_ap_mdev_get_locks - acquire the locks required to assign/unassign AP
+ *                adapters, domains and control domains for an mdev in
+ *                the proper locking order.
+ *
+ * @matrix_mdev: the matrix mediated device object
+ */
+static void vfio_ap_mdev_get_locks(struct ap_matrix_mdev *matrix_mdev)

drivers/s390/crypto/vfio_ap_ops.c +5 -0
+{
+    /* Lock the mutex required to access the KVM guest's state */
+    mutex_lock(&matrix_dev->guests_lock);
+
+    /* If a KVM guest is running, lock the mutex required to 

plug/unplug the
+     * AP devices passed through to the guest
+     */
+    if (matrix_mdev->kvm)
+        mutex_lock(&matrix_mdev->kvm->lock);

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_mdev_link_adapter(struct ap_matrix_mdev *matrix_mdev, · line 664

Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +4 -0
+
+    /* The lock required to access the mdev's state */
+    mutex_lock(&matrix_dev->mdevs_lock);
+}

Simplifying the cdoe, and removing duplication by moving the locking code to a function is probably a good thing. But I don't feel like this belongs to this particular patch. In general, a patch should only do one thing, and ideally that one thing should be as small as reasonably possible. This makes the patch easier to read and to review.

Quoted 4 lines

I feel like, as much as possible, you should refactor the locking in a series of patches that are all kept together. Ideally, they would be a patch series completely separate from dynamic ap. After all, this series is already at 18 patches. :)

I'm going to have to disagree, this locking scheme makes no sense outside of this series. It is only necessary because we now update a guest's APCB whenever an adapter, domain or control domain is assigned or unassigned, when a queue device is probed or removed and when the vfio_ap driver is notified that the host's AP configuration has changed.

Quoted 9 lines

Prior to this series, a guest's APCB was updated only when the vfio_ap driver was notified that the KVM pointer was set or cleared, so it was
only necessary to ensure the kvm->lock is taken before the matrix_dev->lock in the functions that handle the VFIO_GROUP_NOTIFY_SET_KVM group notification event. Prior to this, a patch series to introduce the matrix_dev->guests lock would make no sense because it is not needed to enforce the locking order in those functions listed in the previous paragraph because we didn't update the guest's APCB in those functions.

Quoted 6 lines

I don't understand the lock code enough to argue a whole lot here :) But I do still think, at the very least, that your refactoring of the locking into get_locks/put_locks functions really does belong in a separate patch. Refactoring is not directly related to the hotplug/unplug. Also, this is not a minor refactor. This refactoring touches the code all over the place and really just adds noise to this patch. That noise makes it harder to review.

Okay, this is a little different than what you were asking for in your previous review comment in which you suggested the locking patches should be in a series separate from dynamic ap. This I can get on board with; however, we are now talking about increasing the number of patches in this series.

Quoted 47 lines
Quoted 47 lines
Quoted 28 lines

...

Quoted 25 lines

  /**
   * assign_adapter_store - parses the APID from @buf and sets the
   * corresponding bit in the mediated matrix device's APM

drivers/s390/crypto/vfio_ap_ops.c +1 -8
@@ -649,17 +723,9 @@ static ssize_t assign_adapter_store(struct 
 device *dev,
       int ret;
       unsigned long apid;
       DECLARE_BITMAP(apm, AP_DEVICES);
-
       struct ap_matrix_mdev *matrix_mdev = dev_get_drvdata(dev);
   -    mutex_lock(&matrix_dev->guests_lock);
-    mutex_lock(&matrix_dev->mdevs_lock);
-
-    /* If the KVM guest is running, disallow assignment of adapter */
-    if (matrix_mdev->kvm) {
-        ret = -EBUSY;
-        goto done;
-    }
+    vfio_ap_mdev_get_locks(matrix_mdev);
         ret = kstrtoul(buf, 0, &apid);
       if (ret)
@@ -671,8 +737,6 @@ static ssize_t assign_adapter_store(struct 
 device *dev,
       }
         set_bit_inv(apid, matrix_mdev->matrix.apm);
Quoted 8 lines

drivers/s390/crypto/vfio_ap_ops.c static ssize_t assign_adapter_store(struct device *dev, · line 741

Quoted 3 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -2
-    memset(apm, 0, sizeof(apm));
-    set_bit_inv(apid, apm);
         ret = vfio_ap_mdev_validate_masks(matrix_mdev);

It looks like you moved the memset() and set_bit_inv() to be closer to where "apm" is used, namely, the call to vfio_ap_mdev_filter_matrix(). Any reason you cannot move it down under the call to vfio_ap_mdev_link_adapter()? That would get it even closer to where it is used.

I didn't move it to be closer to where it is used, I moved it because it was not necessary to do the memset/set_bit_inv when not necessary to do so. Having said that, it can definitely be moved after the vfio_ap_mdev_link_adapter().

Quoted 5 lines

Also, I think renaming apm to apm_delta or apm_diff makes sense here. After all, it is the difference between the original apm, and the new apm. The new apm has an extra bit for the newly added adapter. Do I have that right? If so, I think renaming the variable will make the code clearer.

Quoted 8 lines
Quoted 4 lines

The purpose of this bitmap is to limit the filtering to the new APID being assigned because there is no need to do filtering of adapters already assigned; so, it is not really a new apm per se. It might be more accurate to call it new_apid or new_apids, although there will only be one bit set in the bitmap.

My main concern here was generic variables names. The "new_apm" will have exactly one
new bit set. That bit is the delta (or the difference) between the previously existing apm, and the new apm, which will be the result of adding in whatever the "apid" bit is. Therefore, it really is a delta, right? This was the basis for my suggestion of the name. Its really not the "new" apm... its the difference between the old and new.

I overlooked the names you suggested - i.e., apm_delta/apm_diff - and for some reason beamed in on "new apm" as if you were suggesting that as the name of the variable. Of the two names, I prefer apm_delta, so I will go with that.

Quoted 3 lines
author maintainer
#30 lore raw

Re: [PATCH v18 12/18] s390/vfio-ap: reset queues after adapter/domain unassignment

In reply to Jason J. Herne
Quoted 81 lines · Jason J. Herne
Quoted 9 lines · Tony Krowiak

When an adapter or domain is unassigned from an mdev providing the AP configuration to a running KVM guest, one or more of the guest's queues may get dynamically removed. Since the removed queues could get re-assigned to another mdev, they need to be reset. So, when an adapter or domain is unassigned from the mdev, the queues that are removed from the guest's AP configuration will be reset.

---

...

Quoted 34 lines

  +static void vfio_ap_unlink_apqn_fr_mdev(struct ap_matrix_mdev *matrix_mdev, +                    unsigned long apid, unsigned long apqi,
+                    struct ap_queue_table *qtable)

drivers/s390/crypto/vfio_ap_ops.c +16 -0
+{
+    struct vfio_ap_queue *q;
+
+    q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi));
+    /* If the queue is assigned to the matrix mdev, unlink it. */
+    if (q)
+        vfio_ap_unlink_queue_fr_mdev(q);
+
+    /* If the queue is assigned to the APCB, store it in @qtable. */
+    if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) &&
+        test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm))
+        hash_add(qtable->queues, &q->mdev_qnode, q->apqn);
+}
+
+/**
+ * vfio_ap_mdev_unlink_adapter - unlink all queues associated with 

unassigned
+ *                 adapter from the matrix mdev to which the
+ *                 adapter was assigned.
+ * @matrix_mdev: the matrix mediated device to which the adapter was assigned.
+ * @apid: the APID of the unassigned adapter.
+ * @qtable: table for storing queues associated with unassigned adapter.
+ */
  static void vfio_ap_mdev_unlink_adapter(struct ap_matrix_mdev *matrix_mdev, -                    unsigned long apid)
+                    unsigned long apid,
+                    struct ap_queue_table *qtable)
  {
      unsigned long apqi;

drivers/s390/crypto/vfio_ap_ops.c static ssize_t assign_adapter_store(struct device *dev, · line 792

Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +4 -0
+
+    for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS)
+        vfio_ap_unlink_apqn_fr_mdev(matrix_mdev, apid, apqi, qtable);
+}

Here is an alternate version of the above two functions that stops the profileration of the qtables variable into vfio_ap_unlink_apqn_fr_mdev. It may seem like a change with no benefit, but it simplifies things a bit and avoids the reader from having to sink three functions deep to find out where qtables is used. This is 100% untested.

static bool vfio_ap_unlink_apqn_fr_mdev(struct ap_matrix_mdev *matrix_mdev, unsigned long apid, unsigned long apqi)
{
    struct vfio_ap_queue *q;

    q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi)); /* If the queue is assigned to the matrix mdev, unlink it. */ if (q) {
        vfio_ap_unlink_queue_fr_mdev(q);
        return true;
    }
    return false;
}

static void vfio_ap_mdev_unlink_adapter(struct ap_matrix_mdev *matrix_mdev, unsigned long apid,
                    struct ap_queue_table *qtable)
{
    unsigned long apqi;
    bool unlinked;

    for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) { unlinked = vfio_ap_unlink_apqn_fr_mdev(matrix_mdev, apid, apqi, qtable);

Quoted 8 lines

        if (unlinked && qtable) {
            if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) && test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) hash_add(qtable->queues, &q->mdev_qnode, q->apqn);
        }
    }
}

This code didn't compile because in the function immediately above, the variable q is not defined nor is it initialized with a value. What I did to fix that is return  the vfio_ap_queue pointer from the vfio_ap_unlink_apqn_fr_mdev function and checked the return value for NULL instead of the boolean:

vfio_ap_queue *q;
...
q = vfio_ap_unlink_apqn_fr_mdev(matrix_mdev, apid, apqi, qtable);
...
if (q && qtable)
...

Quoted 17 lines
Quoted 17 lines

+static void vfio_ap_mdev_hot_unplug_adapter(struct ap_matrix_mdev *matrix_mdev,
+                        unsigned long apid)
+{
+    int bkt;
      struct vfio_ap_queue *q;
+    struct ap_queue_table qtable;
+    hash_init(qtable.queues);
+    vfio_ap_mdev_unlink_adapter(matrix_mdev, apid, &qtable); +
+    if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm)) { +        clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm); +        vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+    }
  -    for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) {

Quoted 8 lines
Quoted 5 lines

-        q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi)); +    vfio_ap_mdev_reset_queues(&qtable);
  -        if (q)
-            vfio_ap_mdev_unlink_queue(q);
+    hash_for_each(qtable.queues, bkt, q, mdev_qnode) {

This comment applies to all instances of btk: What is btk? Can we come up with a more descriptive name?

If you look at the hash_for_each macro, you will see that I used the exact same variable name as the macro does to indicate it is a bucket loop cursor. Since that is a long name I'll go with loop_cursor.

Quoted 7 lines
Quoted 6 lines

+        vfio_ap_unlink_mdev_fr_queue(q);
+        hash_del(&q->mdev_qnode);
      }
  }

...

Quoted 14 lines

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_mdev_unset_kvm(struct · line 1327

Quoted 11 lines
drivers/s390/crypto/vfio_ap_ops.c +3 -2
@@ -1273,9 +1320,9 @@ static void vfio_ap_mdev_unset_kvm(struct 
 ap_matrix_mdev *matrix_mdev,
           mutex_lock(&kvm->lock);
           mutex_lock(&matrix_dev->mdevs_lock);
   -        kvm_arch_crypto_clear_masks(kvm);
-        vfio_ap_mdev_reset_queues(matrix_mdev);
-        kvm_put_kvm(kvm);
+        kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
+        vfio_ap_mdev_reset_queues(&matrix_mdev->qtable);
+        kvm_put_kvm(matrix_mdev->kvm);
           matrix_mdev->kvm = NULL;

I understand changing the call to vfio_ap_mdev_reset_queues, but why are we changing the kvm pointer on the surrounding lines?

In reality, both pointers are one in the same given the two callers pass matrix_mdev->kvm into the function. I'm not sure why that is the case, it is probably a remnant from the commits that fixed the lockdep splat; however, there is no reason other than I've gotten used to retrieving the KVM pointer from the ap_matrix_mdev structure. In reality, there is no reason to pass a 'struct kvm *kvm' into this function, so I'm going to look into that and adjust accordingly.

Quoted 11 lines

drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_reset_queue(struct · line 1378

Quoted 8 lines
drivers/s390/crypto/vfio_ap_ops.c +1 -0
 mutex_unlock(&matrix_dev->mdevs_lock);
@@ -1328,14 +1375,17 @@ static int vfio_ap_mdev_reset_queue(struct 
 vfio_ap_queue *q, unsigned int retry)
         if (!q)
           return 0;
+    q->reset_rc = 0;

This line seems unnecessary. You set q->reset_rc in every single case below, so this 0 will always get overwritten.

Right you are. It is also unnecessary to set q->reset_rc every case when it can be set once right after the call to ap_zapq.

Quoted 40 lines
Quoted 38 lines

  retry_zapq:
      status = ap_zapq(q->apqn);
      switch (status.response_code) {
      case AP_RESPONSE_NORMAL:
          ret = 0;
+        q->reset_rc = status.response_code;
          break;
      case AP_RESPONSE_RESET_IN_PROGRESS:
+        q->reset_rc = status.response_code;
          if (retry--) {
              msleep(20);
              goto retry_zapq;

(quoted diff) +0 -0
@@ -1345,13 +1395,20 @@ static int vfio_ap_mdev_reset_queue(struct 
 vfio_ap_queue *q, unsigned int retry)

      case AP_RESPONSE_Q_NOT_AVAIL:
      case AP_RESPONSE_DECONFIGURED:
      case AP_RESPONSE_CHECKSTOPPED:
-        WARN_ON_ONCE(status.irq_enabled);
+        WARN_ONCE(status.irq_enabled,
+              "PQAP/ZAPQ for %02x.%04x failed with rc=%u while IRQ enabled",
+              AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn), +              status.response_code);
+        q->reset_rc = status.response_code;
          ret = -EBUSY;
          goto free_resources;
      default:
          /* things are really broken, give up */
-        WARN(true, "PQAP/ZAPQ completed with invalid rc (%x)\n", +        WARN(true,
+             "PQAP/ZAPQ for %02x.%04x failed with invalid rc=%u\n", +             AP_QID_CARD(q->apqn), AP_QID_QUEUE(q->apqn), status.response_code);
+        q->reset_rc = status.response_code;
          return -EIO;
      }

...

author maintainer
#31 lore raw

Re: [PATCH v18 12/18] s390/vfio-ap: reset queues after adapter/domain unassignment

snip ...

Quoted 23 lines
Quoted 8 lines
Quoted 7 lines

+ vfio_ap_unlink_mdev_fr_queue(q);
+        hash_del(&q->mdev_qnode);
      }
  }

...

Quoted 14 lines

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_mdev_unset_kvm(struct · line 1327

Quoted 11 lines
drivers/s390/crypto/vfio_ap_ops.c +3 -2
@@ -1273,9 +1320,9 @@ static void vfio_ap_mdev_unset_kvm(struct 
 ap_matrix_mdev *matrix_mdev,
           mutex_lock(&kvm->lock);
           mutex_lock(&matrix_dev->mdevs_lock);
   -        kvm_arch_crypto_clear_masks(kvm);
-        vfio_ap_mdev_reset_queues(matrix_mdev);
-        kvm_put_kvm(kvm);
+        kvm_arch_crypto_clear_masks(matrix_mdev->kvm);
+ vfio_ap_mdev_reset_queues(&matrix_mdev->qtable);
+        kvm_put_kvm(matrix_mdev->kvm);
           matrix_mdev->kvm = NULL;

I understand changing the call to vfio_ap_mdev_reset_queues, but why are we changing the kvm pointer on the surrounding lines?

Quoted 7 lines

In reality, both pointers are one in the same given the two callers pass matrix_mdev->kvm into the function. I'm not sure why that is the case, it is probably a remnant from the commits that fixed the lockdep splat; however, there is no reason other than I've gotten used to retrieving the KVM pointer from the ap_matrix_mdev structure. In reality, there is no reason to pass a 'struct kvm *kvm' into this function, so I'm going to look into that and adjust accordingly.

The 'struct kvm *kvm' parameter was added to the signature of the vfio_ap_mdev_unset_kvm function with the following commit:

86956e70761b (s390/vfio-ap: replace open coded locks for VFIO_GROUP_NOTIFY_SET_KVM notification)

I also noticed the the kernel doc for the vfio_ap_mdev_set_kvm and vfio_ap_mdev_unset_kvm functions still contained a comment that is no longer valid by the following commit:

0cc00c8d4050 (s390/vfio-ap: fix circular lockdep when setting/clearing crypto masks)

I pushed a patch to our devel branch that removes the invalid comment from the two functions and removes the 'struct kvm *kvm' parameter from the vfio_ap_mdev_unset_kvm function. That patch will prereq this series.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
drivers/s390/crypto/vfio_ap_ops.c +1 -0
 mutex_unlock(&matrix_dev->mdevs_lock);
@@ -1328,14 +1375,17 @@ static int vfio_ap_mdev_reset_queue(struct 
 vfio_ap_queue *q, unsigned int retry)
         if (!q)
           return 0;
+    q->reset_rc = 0;
maintainer
#33 lore raw

Re: [PATCH v18 13/18] s390/vfio-ap: implement in-use callback for vfio_ap driver

In reply to Tony Krowiak

On 2/14/22 19:50, Tony Krowiak wrote:
...

Quoted 27 lines
drivers/s390/crypto/vfio_ap_ops.c +15 -2
index e9f7ec6fc6a5..63dfb9b89581 100644
@@ -617,10 +617,32 @@ static int vfio_ap_mdev_verify_no_sharing(unsigned long *mdev_apm,
  	return 0;
  }
 
+/**
+ * vfio_ap_mdev_validate_masks - verify that the APQNs assigned to the mdev are
+ *				 not reserved for the default zcrypt driver and
+ *				 are not assigned to another mdev.
+ *
+ * @matrix_mdev: the mdev to which the APQNs being validated are assigned.
+ *
+ * Return: One of the following values:
+ * o the error returned from the ap_apqn_in_matrix_owned_by_def_drv() function,
+ *   most likely -EBUSY indicating the ap_perms_mutex lock is already held.
+ * o EADDRNOTAVAIL if an APQN assigned to @matrix_mdev is reserved for the
+ *		   zcrypt default driver.
+ * o EADDRINUSE if an APQN assigned to @matrix_mdev is assigned to another mdev
+ * o A zero indicating validation succeeded.
+ */
  static int vfio_ap_mdev_validate_masks(struct ap_matrix_mdev *matrix_mdev)
  {
-	if (ap_apqn_in_matrix_owned_by_def_drv(matrix_mdev->matrix.apm,
-					       matrix_mdev->matrix.aqm))

drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_verify_no_sharing(unsigned long *mdev_apm, · line 646

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +9 -0
+	int ret;
+
+	ret = ap_apqn_in_matrix_owned_by_def_drv(matrix_mdev->matrix.apm,
+						 matrix_mdev->matrix.aqm);
+
+	if (ret < 0)
+		return ret;
+
+	if (ret == 1)
  		return -EADDRNOTAVAIL;

I took a look at ap_apqn_in_matrix_owned_by_def_drv(). It appears that this function can only ever return 0 or 1. This patch is changed to watch for a negative return value from ap_apqn_in_matrix_owned_by_def_drv(). Am I missing something?

Signature · 1 line
-- Jason J. Herne ([email protected])
maintainer Reviewed-by on v18 14/18
#36 lore raw

Re: [PATCH v18 14/18] s390/vfio-ap: sysfs attribute to display the guest's matrix

In reply to Tony Krowiak
Quoted 8 lines · Tony Krowiak

The matrix of adapters and domains configured in a guest's APCB may differ from the matrix of adapters and domains assigned to the matrix mdev, so this patch introduces a sysfs attribute to display the matrix of adapters and domains that are or will be assigned to the APCB of a guest that is or will be using the matrix mdev. For a matrix mdev denoted by $uuid, the guest matrix can be displayed as follows:

cat /sys/devices/vfio_ap/matrix/$uuid/guest_matrix

My OCD wants you to name this matrix_guest instead of guest_matrix. Simply because then "matrix" and "matrix_guest" will be grouped together when doing an ls on the parent directory. As a system admin, its the little things that make the difference :) Please consider... though I won't withhold an R-b for it.

Signature · 1 line
-- Jason J. Herne ([email protected])
author maintainer
#34 lore raw

Re: [PATCH v18 13/18] s390/vfio-ap: implement in-use callback for vfio_ap driver

In reply to Jason J. Herne
Quoted 41 lines · Jason J. Herne

On 2/14/22 19:50, Tony Krowiak wrote:
...

Quoted 39 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -0
index e9f7ec6fc6a5..63dfb9b89581 100644
@@ -617,10 +617,32 @@ static int 
 vfio_ap_mdev_verify_no_sharing(unsigned long *mdev_apm,
       return 0;
   }

  +/**
+ * vfio_ap_mdev_validate_masks - verify that the APQNs assigned to the mdev are
+ *                 not reserved for the default zcrypt driver and
+ *                 are not assigned to another mdev.
+ *
+ * @matrix_mdev: the mdev to which the APQNs being validated are assigned.
+ *
+ * Return: One of the following values:
+ * o the error returned from the ap_apqn_in_matrix_owned_by_def_drv() function,
+ *   most likely -EBUSY indicating the ap_perms_mutex lock is already held.
+ * o EADDRNOTAVAIL if an APQN assigned to @matrix_mdev is reserved for the
+ *           zcrypt default driver.
+ * o EADDRINUSE if an APQN assigned to @matrix_mdev is assigned to another mdev
+ * o A zero indicating validation succeeded.
+ */
  static int vfio_ap_mdev_validate_masks(struct ap_matrix_mdev *matrix_mdev) {
-    if (ap_apqn_in_matrix_owned_by_def_drv(matrix_mdev->matrix.apm, -                           matrix_mdev->matrix.aqm))
+    int ret;

drivers/s390/crypto/vfio_ap_ops.c +5 -0
+
+    ret = ap_apqn_in_matrix_owned_by_def_drv(matrix_mdev->matrix.apm,
+                         matrix_mdev->matrix.aqm);
+
+    if (ret < 0)
Quoted 8 lines
Quoted 4 lines

+        return ret;
+
+    if (ret == 1)
          return -EADDRNOTAVAIL;

I took a look at ap_apqn_in_matrix_owned_by_def_drv(). It appears that this function can only ever return 0 or 1. This patch is changed to watch for a negative return value from ap_apqn_in_matrix_owned_by_def_drv(). Am I missing something?

That's odd, careless error, I'll fix it.

Quoted 2 lines
author maintainer
#37 lore raw

Re: [PATCH v18 14/18] s390/vfio-ap: sysfs attribute to display the guest's matrix

In reply to Jason J. Herne
Quoted 8 lines · Jason J. Herne
Quoted 7 lines · Tony Krowiak

The matrix of adapters and domains configured in a guest's APCB may differ from the matrix of adapters and domains assigned to the matrix mdev, so this patch introduces a sysfs attribute to display the matrix of adapters and domains that are or will be assigned to the APCB of a guest that is or will be using the matrix mdev. For a matrix mdev denoted by $uuid, the guest matrix can be displayed as follows:

Quoted 6 lines
Quoted 1 line

    cat /sys/devices/vfio_ap/matrix/$uuid/guest_matrix

My OCD wants you to name this matrix_guest instead of guest_matrix. Simply because then "matrix" and "matrix_guest" will be grouped together when doing an ls on the parent directory. As a system admin, its the little things that make the difference :) Please consider... though I won't withhold an R-b for it.

I am going to leave it as guest_matrix for two reasons:
1. To me, the name matrix_guest reads like the guest belongs to the matrix rather than the matrix belonging to the guest.
2. Changing it will require changes to all of the automated test cases that check sysfs to determine whether the attribute is present and I don't think the juice is worth the squeeze.

Quoted 3 lines
maintainer
#39 lore raw

Re: [PATCH v18 15/18] s390/vfio-ap: handle config changed and scan complete notification

In reply to Tony Krowiak

On 2/14/22 19:50, Tony Krowiak wrote:
...

Quoted 11 lines
drivers/s390/crypto/vfio_ap_ops.c +1 -4
@@ -790,13 +788,17 @@ static void vfio_ap_unlink_apqn_fr_mdev(struct ap_matrix_mdev *matrix_mdev,
 
  	q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi));
  	/* If the queue is assigned to the matrix mdev, unlink it. */
-	if (q)
+	if (q) {
  		vfio_ap_unlink_queue_fr_mdev(q);
 
-	/* If the queue is assigned to the APCB, store it in @qtable. */
-	if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) &&
-	    test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm))

drivers/s390/crypto/vfio_ap_ops.c static void vfio_ap_unlink_apqn_fr_mdev(struct ap_matrix_mdev *matrix_mdev, · line 802

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +8 -1
-		hash_add(qtable->queues, &q->mdev_qnode, q->apqn);
+		/* If the queue is assigned to the APCB, store it in @qtable. */
+		if (qtable) {
+			if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) &&
+			    test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm))
+				hash_add(qtable->queues, &q->mdev_qnode,
+					 q->apqn);
+		}
+	}
  }

This appears to be an unrelated change. Does this belong in this patch?

drivers/s390/crypto/vfio_ap_ops.c static const struct attribute_group *vfio_ap_mdev_attr_groups[] = { · line 1276

Quoted 7 lines
drivers/s390/crypto/vfio_ap_ops.c +1 -1
  /**
@@ -1271,7 +1273,7 @@ static const struct attribute_group *vfio_ap_mdev_attr_groups[] = {
   * @matrix_mdev: a mediated matrix device
   * @kvm: reference to KVM instance
   *
- * Note: The matrix_dev->lock must be taken prior to calling
+ * Note: The matrix_dev->mdevs_lock must be taken prior to calling

This also seems to be unrelated.

drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_iommu_notifier(struct notifier_block *nb, · line 1360

Quoted 9 lines
drivers/s390/crypto/vfio_ap_ops.c +1 -1
   * this function; however, the lock will be temporarily released while the
   * guest's AP configuration is set to avoid a potential lockdep splat.
   * The kvm->lock is taken to set the guest's AP configuration which, under
@@ -1355,7 +1357,7 @@ static int vfio_ap_mdev_iommu_notifier(struct notifier_block *nb,
   * @matrix_mdev: a matrix mediated device
   * @kvm: the pointer to the kvm structure being unset.
   *
- * Note: The matrix_dev->lock must be taken prior to calling
+ * Note: The matrix_dev->mdevs_lock must be taken prior to calling

Same here.

Quoted 57 lines
drivers/s390/crypto/vfio_ap_ops.c +29 -6
   * this function; however, the lock will be temporarily released while the
   * guest's AP configuration is cleared to avoid a potential lockdep splat.
   * The kvm->lock is taken to clear the guest's AP configuration which, under
@@ -1708,6 +1710,27 @@ static void vfio_ap_mdev_put_qlocks(struct ap_matrix_mdev *matrix_mdev)
  	mutex_unlock(&matrix_dev->guests_lock);
  }
 
+static bool vfio_ap_mdev_do_filter_matrix(struct ap_matrix_mdev *matrix_mdev,
+					  struct vfio_ap_queue *q)
+{
+	unsigned long apid = AP_QID_CARD(q->apqn);
+	unsigned long apqi = AP_QID_QUEUE(q->apqn);
+
+	/*
+	 * If the queue is being probed because its APID or APQI is in the
+	 * process of being added to the host's AP configuration, then we don't
+	 * want to filter the matrix now as the filtering will be done after
+	 * the driver is notified that the AP bus scan operation has completed
+	 * (see the vfio_ap_on_scan_complete callback function).
+	 */
+	if (test_bit_inv(apid, matrix_mdev->apm_add) ||
+	    test_bit_inv(apqi, matrix_mdev->aqm_add))
+		return false;
+
+
+	return true;
+}
+
  int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
  {
  	struct vfio_ap_queue *q;
@@ -1725,10 +1748,15 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
  		vfio_ap_mdev_link_queue(matrix_mdev, q);
  		memset(apm, 0, sizeof(apm));
  		set_bit_inv(AP_QID_CARD(q->apqn), apm);
-		if (vfio_ap_mdev_filter_matrix(apm, q->matrix_mdev->matrix.aqm,
-					       q->matrix_mdev))
-			vfio_ap_mdev_hotplug_apcb(q->matrix_mdev);
+
+		if (vfio_ap_mdev_do_filter_matrix(q->matrix_mdev, q)) {
+			if (vfio_ap_mdev_filter_matrix(apm,
+						q->matrix_mdev->matrix.aqm,
+						q->matrix_mdev))
+				vfio_ap_mdev_hotplug_apcb(q->matrix_mdev);
+		}
  	}
+
  	dev_set_drvdata(&apdev->device, q);
  	vfio_ap_mdev_put_qlocks(matrix_mdev);
 
@@ -1783,10 +1811,15 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev)
 
  		apid = AP_QID_CARD(q->apqn);
  		apqi = AP_QID_QUEUE(q->apqn);
-		if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) &&
-		    test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) {
-			clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm);

drivers/s390/crypto/vfio_ap_ops.c void vfio_ap_mdev_remove_queue(struct ap_device *apdev) · line 1822

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +9 -1
-			vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+
+		/*
+		 * If the queue is assigned to the guest's APCB, then remove
+		 * the adapter's APID from the APCB and hot it into the guest.
+		 */
+		if (test_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm) &&
+		    test_bit_inv(apqi, q->matrix_mdev->shadow_apcb.aqm)) {
+			clear_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm);
+			vfio_ap_mdev_hotplug_apcb(q->matrix_mdev);

It looks like this a bug fix unrelated to this patch...?

Quoted 13 lines
drivers/s390/crypto/vfio_ap_ops.c +8 -0
@@ -1842,3 +1875,267 @@ int vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm)
 
  	return ret;
  }
+
+/**
+ * vfio_ap_mdev_hot_unplug_cfg - hot unplug the adapters, domains and control
+ *				 domains that have been removed from the host's
+ *				 AP configuration from a guest.
+ *
+ * @guest: the guest
+ * @aprem: the adapters that have been removed from the host's AP configuration

drivers/s390/crypto/vfio_ap_ops.c int vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm) · line 1895

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +10 -0
+ * @aqrem: the domains that have been removed from the host's AP configuration
+ * @cdrem: the control domains that have been removed from the host's AP
+ *	   configuration.
+ */
+static void vfio_ap_mdev_hot_unplug_cfg(struct ap_matrix_mdev *matrix_mdev,
+					unsigned long *aprem,
+					unsigned long *aqrem,
+					unsigned long *cdrem)
+{
+	bool do_hotplug = false;

__bitmap_andnot() returns an int, so I think you should use an int here.

drivers/s390/crypto/vfio_ap_ops.c int vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm) · line 1900

Quoted 4 lines
drivers/s390/crypto/vfio_ap_ops.c +4 -0
+	if (!bitmap_empty(aprem, AP_DEVICES)) {
+		do_hotplug |= bitmap_andnot(matrix_mdev->shadow_apcb.apm,
+					    matrix_mdev->shadow_apcb.apm,
+					    aprem, AP_DEVICES);

Also, replace the |= with an = here. This is the first assignment so no need to do a logical OR as there is no pre-existing data to preserve.

Quoted 30 lines
drivers/s390/crypto/vfio_ap_ops.c +30 -0
+	}
+
+	if (!bitmap_empty(aqrem, AP_DOMAINS)) {
+		do_hotplug |= bitmap_andnot(matrix_mdev->shadow_apcb.aqm,
+					    matrix_mdev->shadow_apcb.aqm,
+					    aqrem, AP_DEVICES);
+	}
+
+	if (!bitmap_empty(cdrem, AP_DOMAINS))
+		do_hotplug |= bitmap_andnot(matrix_mdev->shadow_apcb.adm,
+					    matrix_mdev->shadow_apcb.adm,
+					    cdrem, AP_DOMAINS);
+
+	if (do_hotplug)
+		vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+}
+
+/**
+ * vfio_ap_mdev_cfg_remove - determines which guests are using the adapters,
+ *			     domains and control domains that have been removed
+ *			     from the host AP configuration and unplugs them
+ *			     from those guests.
+ *
+ * @ap_remove:	bitmap specifying which adapters have been removed from the host
+ *		config.
+ * @aq_remove:	bitmap specifying which domains have been removed from the host
+ *		config.
+ * @cd_remove:	bitmap specifying which control domains have been removed from
+ *		the host config.
+ */

...

drivers/s390/crypto/vfio_ap_ops.c int vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm) · line 1975

Quoted 9 lines
drivers/s390/crypto/vfio_ap_ops.c +9 -0
+/**
+ * vfio_ap_mdev_on_cfg_remove - responds to the removal of adapters, domains and
+ *				control domains from the host AP configuration
+ *				by unplugging them from the guests that are
+ *				using them.
+ */
+static void vfio_ap_mdev_on_cfg_remove(void)
+{
+	int ap_remove, aq_remove, cd_remove;

These can all be replaced with a single variable, just like you did with do_add.

Quoted 20 lines
drivers/s390/crypto/vfio_ap_ops.c +20 -0
+	DECLARE_BITMAP(aprem, AP_DEVICES);
+	DECLARE_BITMAP(aqrem, AP_DOMAINS);
+	DECLARE_BITMAP(cdrem, AP_DOMAINS);
+	unsigned long *cur_apm, *cur_aqm, *cur_adm;
+	unsigned long *prev_apm, *prev_aqm, *prev_adm;
+
+	cur_apm = (unsigned long *)matrix_dev->config_info.apm;
+	cur_aqm = (unsigned long *)matrix_dev->config_info.aqm;
+	cur_adm = (unsigned long *)matrix_dev->config_info.adm;
+	prev_apm = (unsigned long *)matrix_dev->config_info_prev.apm;
+	prev_aqm = (unsigned long *)matrix_dev->config_info_prev.aqm;
+	prev_adm = (unsigned long *)matrix_dev->config_info_prev.adm;
+
+	ap_remove = bitmap_andnot(aprem, prev_apm, cur_apm, AP_DEVICES);
+	aq_remove = bitmap_andnot(aqrem, prev_aqm, cur_aqm, AP_DOMAINS);
+	cd_remove = bitmap_andnot(cdrem, prev_adm, cur_adm, AP_DOMAINS);
+
+	if (ap_remove || aq_remove || cd_remove)
+		vfio_ap_mdev_cfg_remove(aprem, aqrem, cdrem);
+}

...

Quoted 6 lines
drivers/s390/crypto/vfio_ap_ops.c +6 -0
+/**
+ * vfio_ap_on_cfg_changed - handles notification of changes to the host AP
+ *			    configuration.
+ *
+ * @new_config_info: the new host AP configuration
+ * @old_config_info: the previous host AP configuration

drivers/s390/crypto/vfio_ap_ops.c int vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm) · line 2074

Quoted 10 lines
drivers/s390/crypto/vfio_ap_ops.c +10 -0
+ */
+void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info,
+			    struct ap_config_info *old_config_info)
+{
+	mutex_lock(&matrix_dev->guests_lock);
+
+	memcpy(&matrix_dev->config_info_prev, old_config_info,
+		       sizeof(struct ap_config_info));
+	memcpy(&matrix_dev->config_info, new_config_info,
+	       sizeof(struct ap_config_info));

Why are we storing old_config_info in the matrix_dev? It appears to only be used within the functions called from right here. Why not just pass it as an argument?

drivers/s390/crypto/vfio_ap_ops.c int vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm) · line 2079

Quoted 5 lines
drivers/s390/crypto/vfio_ap_ops.c +5 -0
+	vfio_ap_mdev_on_cfg_remove();
+	vfio_ap_mdev_on_cfg_add();
+
+	mutex_unlock(&matrix_dev->guests_lock);
+}

Here is an idea to restructure things... consider combining logic from vfio_ap_mdev_on_cfg_remove and vfio_ap_mdev_on_cfg_add with vfio_ap_on_cfg_changed. This makes vfio_ap_on_cfg_changed() longer but it eliminates some duplicated code and gets rid of both vfio_ap_mdev_on_cfg_remove and vfio_ap_mdev_on_cfg_add.
note: Untested... :)

void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info,
struct ap_config_info *old_config_info)

{
	DECLARE_BITMAP(changed_adapters, AP_DEVICES);
	DECLARE_BITMAP(changed_domains, AP_DOMAINS);
	DECLARE_BITMAP(changed_cdoms, AP_DOMAINS);
	unsigned long *cur_apm, *cur_aqm, *cur_adm;
	unsigned long *prev_apm, *prev_aqm, *prev_adm;

	cur_apm = (unsigned long *)new_config_info->apm;
	cur_aqm = (unsigned long *)new_config_info->aqm;
	cur_adm = (unsigned long *)new_config_info->adm;
	prev_apm = (unsigned long *)old_config_info->apm;
	prev_aqm = (unsigned long *)old_config_info->.aqm;
	prev_adm = (unsigned long *)old_config_info->adm;

	mutex_lock(&matrix_dev->guests_lock);
	
	/* Handle host configuration removals */
	bitmap_andnot(changed_adapters, prev_apm, cur_apm, AP_DEVICES);
	bitmap_andnot(changed_domains, prev_aqm, cur_aqm, AP_DOMAINS);
	bitmap_andnot(changed_cdoms, prev_adm, cur_adm, AP_DOMAINS);

	if (changed_adapters || changed_domains || changed_cdoms)
		vfio_ap_mdev_cfg_remove(aprem, aqrem, cdrem);

	bitmap_clear(changed_adapters, 0, AP_DEVICES);
	bitmap_clear(changed_domains, 0, AP_DOMAINS);
	bitmap_clear(changed_cdoms, 0, AP_DOMAINS);

	/* Handle host configuration additions */
	bitmap_andnot(changed_adapters, cur_apm, prev_apm, AP_DEVICES);
	bitmap_andnot(changed_domains, cur_aqm, prev_aqm, AP_DOMAINS);
	bitmap_andnot(changed_cdoms, cur_adm, prev_adm, AP_DOMAINS);

	if (changed_adapters || changed_domains || changed_cdoms) {
		vfio_ap_mdev_cfg_add(apm_add, aqm_add, adm_add);
	}
	mutex_unlock(&matrix_dev->guests_lock);
}

Not sure if where I put the locking is 100% correct. We could do a lock/unlock around each call to vfio_ap_mdev_cfg_{remove|add} but there is probablyt no point to that, right?

Also, a side effect of these changes is that matrix_dev->config_info is no longer updated. I guess we either would need to update it here or update it wherever it was originally updated before this patrch.

Signature · 1 line
-- Jason J. Herne ([email protected])
author maintainer
#40 lore raw

Re: [PATCH v18 15/18] s390/vfio-ap: handle config changed and scan complete notification

Quoted 17 lines · Jason J. Herne

On 2/14/22 19:50, Tony Krowiak wrote:
...

Quoted 15 lines
drivers/s390/crypto/vfio_ap_ops.c +1 -1
@@ -790,13 +788,17 @@ static void vfio_ap_unlink_apqn_fr_mdev(struct 
 ap_matrix_mdev *matrix_mdev,
         q = vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi));
       /* If the queue is assigned to the matrix mdev, unlink it. */
-    if (q)
+    if (q) {
           vfio_ap_unlink_queue_fr_mdev(q);

  -    /* If the queue is assigned to the APCB, store it in @qtable. */ -    if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) && -        test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm))
-        hash_add(qtable->queues, &q->mdev_qnode, q->apqn); +        /* If the queue is assigned to the APCB, store it in @qtable. */
+        if (qtable) {
+            if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) &&

Quoted 8 lines
Quoted 6 lines

+                test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) +                hash_add(qtable->queues, &q->mdev_qnode, +                     q->apqn);
+        }
+    }
  }

This appears to be an unrelated change. Does this belong in this patch?

No, that does not belong in this patch, it belongs in patch
[PATCH v18 12/18] s390/vfio-ap: reset queues after adapter/domain unassignment.

Quoted 12 lines

drivers/s390/crypto/vfio_ap_ops.c static const struct attribute_group · line 1277

Quoted 10 lines

  /**

drivers/s390/crypto/vfio_ap_ops.c +1 -1
@@ -1271,7 +1273,7 @@ static const struct attribute_group 
 *vfio_ap_mdev_attr_groups[] = {
    * @matrix_mdev: a mediated matrix device
    * @kvm: reference to KVM instance
    *
- * Note: The matrix_dev->lock must be taken prior to calling
+ * Note: The matrix_dev->mdevs_lock must be taken prior to calling

This also seems to be unrelated.

The change of the name of the matrix_dev->lock to matrix_dev->mdevs_lock was done in [PATCH v18 09/18] s390/vfio-ap: introduce new mutex to control access to the KVM pointer; however, this comment is not valid. A commit has been pushed for upstream merge which includes removal of this comment.

Quoted 14 lines

drivers/s390/crypto/vfio_ap_ops.c static int vfio_ap_mdev_iommu_notifier(struct · line 1361

Quoted 12 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -0
    * this function; however, the lock will be temporarily released while the
    * guest's AP configuration is set to avoid a potential lockdep splat.

   * The kvm->lock is taken to set the guest's AP configuration which, under

drivers/s390/crypto/vfio_ap_ops.c +1 -1
@@ -1355,7 +1357,7 @@ static int vfio_ap_mdev_iommu_notifier(struct 
 notifier_block *nb,
    * @matrix_mdev: a matrix mediated device
    * @kvm: the pointer to the kvm structure being unset.
    *
- * Note: The matrix_dev->lock must be taken prior to calling
+ * Note: The matrix_dev->mdevs_lock must be taken prior to calling

Same here.

Ditto

Quoted 64 lines
Quoted 64 lines
drivers/s390/crypto/vfio_ap_ops.c +0 -0
    * this function; however, the lock will be temporarily released while the
    * guest's AP configuration is cleared to avoid a potential lockdep splat.

   * The kvm->lock is taken to clear the guest's AP configuration which, under

drivers/s390/crypto/vfio_ap_ops.c +8 -0
@@ -1708,6 +1710,27 @@ static void vfio_ap_mdev_put_qlocks(struct 
 ap_matrix_mdev *matrix_mdev)
       mutex_unlock(&matrix_dev->guests_lock);
   }
   +static bool vfio_ap_mdev_do_filter_matrix(struct ap_matrix_mdev *matrix_mdev,
+                      struct vfio_ap_queue *q)
+{
+    unsigned long apid = AP_QID_CARD(q->apqn);
+    unsigned long apqi = AP_QID_QUEUE(q->apqn);
+
+    /*
+     * If the queue is being probed because its APID or APQI is in the
+     * process of being added to the host's AP configuration, then 

we don't
+     * want to filter the matrix now as the filtering will be done after
+     * the driver is notified that the AP bus scan operation has completed
+     * (see the vfio_ap_on_scan_complete callback function). +     */
+    if (test_bit_inv(apid, matrix_mdev->apm_add) ||
+        test_bit_inv(apqi, matrix_mdev->aqm_add))
+        return false;

drivers/s390/crypto/vfio_ap_ops.c +5 -0
+
+
+    return true;
+}
+

  int vfio_ap_mdev_probe_queue(struct ap_device *apdev) {
      struct vfio_ap_queue *q;

drivers/s390/crypto/vfio_ap_ops.c +8 -3
@@ -1725,10 +1748,15 @@ int vfio_ap_mdev_probe_queue(struct ap_device 
 *apdev)
           vfio_ap_mdev_link_queue(matrix_mdev, q);
           memset(apm, 0, sizeof(apm));
           set_bit_inv(AP_QID_CARD(q->apqn), apm);
-        if (vfio_ap_mdev_filter_matrix(apm, q->matrix_mdev->matrix.aqm,
-                           q->matrix_mdev))
-            vfio_ap_mdev_hotplug_apcb(q->matrix_mdev);
+
+        if (vfio_ap_mdev_do_filter_matrix(q->matrix_mdev, q)) {
+            if (vfio_ap_mdev_filter_matrix(apm,
+                        q->matrix_mdev->matrix.aqm,
+                        q->matrix_mdev))
+                vfio_ap_mdev_hotplug_apcb(q->matrix_mdev);
+        }
       }
+
       dev_set_drvdata(&apdev->device, q);
       vfio_ap_mdev_put_qlocks(matrix_mdev);

  @@ -1783,10 +1811,15 @@ void vfio_ap_mdev_remove_queue(struct ap_device *apdev) apid = AP_QID_CARD(q->apqn);
          apqi = AP_QID_QUEUE(q->apqn);
-        if (test_bit_inv(apid, matrix_mdev->shadow_apcb.apm) &&
-            test_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm)) {
-            clear_bit_inv(apid, matrix_mdev->shadow_apcb.apm);
-            vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+
+        /*
+         * If the queue is assigned to the guest's APCB, then remove

Quoted 8 lines
Quoted 6 lines

+         * the adapter's APID from the APCB and hot it into the guest. +         */
+        if (test_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm) && +            test_bit_inv(apqi, q->matrix_mdev->shadow_apcb.aqm)) { +            clear_bit_inv(apid, q->matrix_mdev->shadow_apcb.apm); +            vfio_ap_mdev_hotplug_apcb(q->matrix_mdev);

It looks like this a bug fix unrelated to this patch...?

It's not a bug fix, but it is unnecessary. Both what was removed and the code added are fine, but the former makes more sense. I have no idea how this happened other than the code was accidentally changed while merging code during one of the multitude of rebases undertaken while developing this series.

Quoted 21 lines
Quoted 21 lines

  @@ -1842,3 +1875,267 @@ int vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm) return ret;
  }

drivers/s390/crypto/vfio_ap_ops.c +3 -0
+
+/**
+ * vfio_ap_mdev_hot_unplug_cfg - hot unplug the adapters, domains 

and control
+ *                 domains that have been removed from the host's
+ *                 AP configuration from a guest.
+ *
+ * @guest: the guest
+ * @aprem: the adapters that have been removed from the host's AP configuration
+ * @aqrem: the domains that have been removed from the host's AP configuration
+ * @cdrem: the control domains that have been removed from the host's AP
+ *       configuration.
+ */
+static void vfio_ap_mdev_hot_unplug_cfg(struct ap_matrix_mdev

Quoted 8 lines
Quoted 6 lines

*matrix_mdev,
+                    unsigned long *aprem,
+                    unsigned long *aqrem,
+                    unsigned long *cdrem)
+{
+    bool do_hotplug = false;

__bitmap_andnot() returns an int, so I think you should use an int here.

Okay

Quoted 9 lines
Quoted 6 lines

+    if (!bitmap_empty(aprem, AP_DEVICES)) {
+        do_hotplug |= bitmap_andnot(matrix_mdev->shadow_apcb.apm, +                        matrix_mdev->shadow_apcb.apm, +                        aprem, AP_DEVICES);

Also, replace the |= with an = here. This is the first assignment so no need to do a logical OR as there is no pre-existing data to preserve.

We don't know which, if any, of the bitmaps (aprem, aqrem, cdrem) passed to this function contain set bits, so the |= is necessary because this code block won't be be executed if aprem is empty. That is why the do_hotplug variable is initialized to false when it is declared (i.e., each of the bitmaps might be empty).

Quoted 37 lines
Quoted 36 lines

+    }

drivers/s390/crypto/vfio_ap_ops.c +18 -0
+
+    if (!bitmap_empty(aqrem, AP_DOMAINS)) {
+        do_hotplug |= bitmap_andnot(matrix_mdev->shadow_apcb.aqm,
+                        matrix_mdev->shadow_apcb.aqm,
+                        aqrem, AP_DEVICES);
+    }
+
+    if (!bitmap_empty(cdrem, AP_DOMAINS))
+        do_hotplug |= bitmap_andnot(matrix_mdev->shadow_apcb.adm,
+                        matrix_mdev->shadow_apcb.adm,
+                        cdrem, AP_DOMAINS);
+
+    if (do_hotplug)
+        vfio_ap_mdev_hotplug_apcb(matrix_mdev);
+}
+
+/**
+ * vfio_ap_mdev_cfg_remove - determines which guests are using the 

adapters,
+ *                 domains and control domains that have been removed
+ *                 from the host AP configuration and unplugs them
+ *                 from those guests.
+ *
+ * @ap_remove:    bitmap specifying which adapters have been removed from the host
+ *        config.
+ * @aq_remove:    bitmap specifying which domains have been removed from the host
+ *        config.
+ * @cd_remove:    bitmap specifying which control domains have been removed from
+ *        the host config.
+ */

...

Quoted 12 lines

drivers/s390/crypto/vfio_ap_ops.c int vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm) · line 1975

Quoted 10 lines

+/**
+ * vfio_ap_mdev_on_cfg_remove - responds to the removal of adapters, domains and
+ *                control domains from the host AP configuration
+ *                by unplugging them from the guests that are
+ *                using them.
+ */

drivers/s390/crypto/vfio_ap_ops.c +3 -0
+static void vfio_ap_mdev_on_cfg_remove(void)
+{
+    int ap_remove, aq_remove, cd_remove;

These can all be replaced with a single variable, just like you did with do_add.

True, consider it done.

Quoted 35 lines
Quoted 21 lines

+    DECLARE_BITMAP(aprem, AP_DEVICES);
+    DECLARE_BITMAP(aqrem, AP_DOMAINS);
+    DECLARE_BITMAP(cdrem, AP_DOMAINS);
+    unsigned long *cur_apm, *cur_aqm, *cur_adm; +    unsigned long *prev_apm, *prev_aqm, *prev_adm;

drivers/s390/crypto/vfio_ap_ops.c +15 -0
+
+    cur_apm = (unsigned long *)matrix_dev->config_info.apm;
+    cur_aqm = (unsigned long *)matrix_dev->config_info.aqm;
+    cur_adm = (unsigned long *)matrix_dev->config_info.adm;
+    prev_apm = (unsigned long *)matrix_dev->config_info_prev.apm;
+    prev_aqm = (unsigned long *)matrix_dev->config_info_prev.aqm;
+    prev_adm = (unsigned long *)matrix_dev->config_info_prev.adm;
+
+    ap_remove = bitmap_andnot(aprem, prev_apm, cur_apm, AP_DEVICES);
+    aq_remove = bitmap_andnot(aqrem, prev_aqm, cur_aqm, AP_DOMAINS);
+    cd_remove = bitmap_andnot(cdrem, prev_adm, cur_adm, AP_DOMAINS);
+
+    if (ap_remove || aq_remove || cd_remove)
+        vfio_ap_mdev_cfg_remove(aprem, aqrem, cdrem);
+}

...

Quoted 13 lines

+/**
+ * vfio_ap_on_cfg_changed - handles notification of changes to the host AP
+ *                configuration.
+ *
+ * @new_config_info: the new host AP configuration
+ * @old_config_info: the previous host AP configuration
+ */

drivers/s390/crypto/vfio_ap_ops.c +5 -0
+void vfio_ap_on_cfg_changed(struct ap_config_info *new_config_info,
+                struct ap_config_info *old_config_info)
+{
+    mutex_lock(&matrix_dev->guests_lock);
+
Quoted 8 lines
Quoted 4 lines

+    memcpy(&matrix_dev->config_info_prev, old_config_info, +               sizeof(struct ap_config_info));
+    memcpy(&matrix_dev->config_info, new_config_info, +           sizeof(struct ap_config_info));

Why are we storing old_config_info in the matrix_dev? It appears to only be used within the functions called from right here. Why not just pass it as an argument?

At the time I thought it might be valuable to keep around. I suppose it can be passed to the functions that need it.

Quoted 14 lines

drivers/s390/crypto/vfio_ap_ops.c int vfio_ap_mdev_resource_in_use(unsigned long *apm, unsigned long *aqm) · line 2079

Quoted 7 lines

+    vfio_ap_mdev_on_cfg_remove();
+    vfio_ap_mdev_on_cfg_add();

drivers/s390/crypto/vfio_ap_ops.c +3 -0
+
+    mutex_unlock(&matrix_dev->guests_lock);
+}

Here is an idea to restructure things... consider combining logic from vfio_ap_mdev_on_cfg_remove and vfio_ap_mdev_on_cfg_add with vfio_ap_on_cfg_changed. This makes vfio_ap_on_cfg_changed() longer but it eliminates some duplicated code and gets rid of both vfio_ap_mdev_on_cfg_remove and vfio_ap_mdev_on_cfg_add.
note: Untested... :)

It seems like a lot of busy work for little gain. I prefer shorter functions that perform a specific task. Since we're solely in the realm of personal preferences, I'm going to leave this function as-is.

maintainer
#43 lore raw

Re: [PATCH v18 17/18] s390/Docs: new doc describing lock usage by the vfio_ap device driver

In reply to Tony Krowiak
Quoted 29 lines · On Mon, 14 Feb 2022 19:50:39 -0500 Tony Krowiak <[email protected]> wrote:

Introduces a new document describing the locks used by the vfio_ap device driver and how to use them so as to avoid lockdep reports and deadlock situations.

---
 Documentation/s390/vfio-ap-locking.rst | 389 +++++++++++++++++++++++++
 1 file changed, 389 insertions(+)
 create mode 100644 Documentation/s390/vfio-ap-locking.rst
Documentation/s390/vfio-ap-locking.rst new file +13 -0
new file mode 100644
index 000000000000..10abbb6d6089
@@ -0,0 +1,389 @@
+======================
+VFIO AP Locks Overview
+======================
+This document describes the locks that are pertinent to the secure operation
+of the vfio_ap device driver. Throughout this document, the following variables
+will be used to denote instances of the structures herein described:
+
+struct ap_matrix_dev *matrix_dev;
+struct ap_matrix_mdev *matrix_mdev;
+struct kvm *kvm;
+
+The Matrix Devices Lock (drivers/s390/crypto/vfio_ap_private.h)
+--------------------------------------------------------------

Documentation/s390/vfio-ap-locking.rst line 23

Quoted 10 lines
Documentation/s390/vfio-ap-locking.rst new file +10 -0
+
+struct ap_matrix_dev {
+	...
+	struct list_head mdev_list;
+	struct mutex mdevs_lock;
+	...
+}
+
+The Matrix Devices Lock (matrix_dev->mdevs_lock) is implemented as a global
+mutex contained within the single instance of struct ap_matrix_dev. This lock

s/single instance of/singleton object of type/

Documentation/s390/vfio-ap-locking.rst line 25

Quoted 2 lines
Documentation/s390/vfio-ap-locking.rst new file +2 -0
+controls access to all fields contained within each matrix_mdev instance under
+the control of the vfio_ap device driver (matrix_dev->mdev_list).

Are there matrix_mdev instances not under the control of the vfio_ap device driver?

(MARK 1)

Documentation/s390/vfio-ap-locking.rst line 28

Quoted 4 lines

This lock must

Documentation/s390/vfio-ap-locking.rst new file +3 -0
+be held while reading from, writing to or using the data from a field contained
+within a matrix_mdev instance representing one of the vfio_ap device driver's
+mediated devices.

This makes it look like for example struct vfio_ap_queue objects are out of scope.

Quoted 23 lines
Documentation/s390/vfio-ap-locking.rst new file +23 -0
+
+The KVM Lock (include/linux/kvm_host.h)
+---------------------------------------
+
+struct kvm {
+	...
+	struct mutex lock;
+	...
+}
+
+The KVM Lock (kvm->lock) controls access to the state data for a KVM guest. This
+lock must be held by the vfio_ap device driver while one or more AP adapters,
+domains or control domains are being plugged into or unplugged from the guest.
+
+The vfio_ap device driver registers a function to be notified when the pointer
+to the kvm instance has been set. The KVM pointer is passed to the handler by
+the notifier and is stored in the in the matrix_mdev instance
+(matrix_mdev->kvm = kvm) containing the state of the mediated device that has
+been passed through to the KVM guest.
+
+The Guests Lock (drivers/s390/crypto/vfio_ap_private.h)
+-----------------------------------------------------------
+

Documentation/s390/vfio-ap-locking.rst line 61

Quoted 10 lines
Documentation/s390/vfio-ap-locking.rst new file +10 -0
+struct ap_matrix_dev {
+	...
+	struct list_head mdev_list;
+	struct mutex guests_lock;
+	...
+}
+
+The Guests Lock (matrix_dev->guests_lock) controls access to the
+matrix_mdev instances (matrix_dev->mdev_list) that represent mediated devices
+that hold the state for the mediated devices that have been passed through to a

Didn't say that access to fields of matrix_mdev instances is controlled by the matrix_dev->mdevs lock at (MARK 1)? How do the two statements mesh?

Quoted 18 lines
Documentation/s390/vfio-ap-locking.rst new file +18 -0
+KVM guest. This lock must be held:
+
+1. To control access to the KVM pointer (matrix_mdev->kvm) while the vfio_ap
+   device driver is using it to plug/unplug AP devices passed through to the KVM
+   guest.
+
+2. To add matrix_mdev instances to or remove them from matrix_dev->mdev_list.
+   This is necessary to ensure the proper locking order when the list is perused
+   to find an ap_matrix_mdev instance for the purpose of plugging/unplugging
+   AP devices passed through to a KVM guest.
+
+   For example, when a queue device is removed from the vfio_ap device driver,
+   if the adapter is passed through to a KVM guest, it will have to be
+   unplugged. In order to figure out whether the adapter is passed through,
+   the matrix_mdev object to which the queue is assigned will have to be
+   found. The KVM pointer (matrix_mdev->kvm) can then be used to determine if
+   the mediated device is passed through (matrix_mdev->kvm != NULL) and if so,
+   to unplug the adapter.

Documentation/s390/vfio-ap-locking.rst line 89

Quoted 10 lines
Documentation/s390/vfio-ap-locking.rst new file +10 -0
+
+It is not necessary to take the Guests Lock to access the KVM pointer if the
+pointer is not used to plug/unplug devices passed through to the KVM guest;
+however, in this case, the Matrix Devices Lock (matrix_dev->mdevs_lock) must be
+held in order to access the KVM pointer since it set and cleared under the
+protection of the Matrix Devices Lock. A case in point is the function that
+handles interception of the PQAP(AQIC) instruction sub-function. This handler
+needs to access the KVM pointer only for the purposes of setting or clearing IRQ
+resources, so only the matrix_dev->mdevs_lock needs to be held.
+

It is very unclear what this lock is actually protecting, and when does it need to be taken.

Quoted 13 lines
Documentation/s390/vfio-ap-locking.rst new file +13 -0
+The PQAP Hook Lock (arch/s390/include/asm/kvm_host.h)
+-----------------------------------------------------
+
+typedef int (*crypto_hook)(struct kvm_vcpu *vcpu);
+
+struct kvm_s390_crypto {
+	...
+	struct rw_semaphore pqap_hook_rwsem;
+	crypto_hook *pqap_hook;
+	...
+};
+
+The PQAP Hook Lock is a r/w semaphore that controls access to the function

Documentation/s390/vfio-ap-locking.rst

Quoted 10 lines
Documentation/s390/vfio-ap-locking.rst new file +10 -0
+pointer of the handler (*kvm->arch.crypto.pqap_hook) to invoke when the
+PQAP(AQIC) instruction sub-function is intercepted by the host. The lock must be
+held in write mode when pqap_hook value is set, and in read mode when the
+pqap_hook function is called.
+
+Locking Order
+-------------
+
+If the various locks are not taken in the proper order, it could potentially
+result in a lockdep splat.

Just in a lockdep splat or in a deadlock?

Quoted 2 lines

The proper order for taking locks depends upon
+the operation taking place,

That sounds very fishy! The whole point of having a locking order is preventing deadlocks if everybody sticks to the locking order. If there are exceptions, i.e. if we violate the locking order, we risk deadlocks.

Documentation/s390/vfio-ap-locking.rst line 119

Quoted 7 lines

but in general the Guests Lock

Documentation/s390/vfio-ap-locking.rst new file +6 -0
+(matrix_dev->guests_lock) must be taken outside of the KVM Lock (kvm->lock)
+which in turn must be taken outside of the Matrix Devices Lock
+(matrix_dev->mdevs_lock).
+
+The following describes the various operations under which the various locks are
+taken, the purpose for taking them and the order in which they must be taken.

It looks a little odd that you describe all the operations here, especially in prosa.

But hey the good thing is, none seem to violate the lock order.

Quoted 270 lines
Documentation/s390/vfio-ap-locking.rst new file +270 -0
+
+* Operations: Setting or clearing the KVM pointer (matrix_mdev->kvm):
+
+  1. PQAP Hook Lock (kvm->arch.crypto.pqap_hook_rwsem):
+
+	This semaphore must be held in write mode while setting or clearing the
+	reference to the function pointer (kvm->arch.crypt.pqap_hook) to call
+	when the PQAP(AQIC) instruction sub-function is intercepted by the host.
+	The function pointer is set when the KVM pointer is being set and
+	cleared when the KVM pointer is being cleared.
+
+  2.Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held while accessing the KVM pointer
+	(matrix_mdev->kvm) to plug/unplug AP devices passed through to the
+	KVM guest
+
+  3. KVM Lock (kvm->lock):
+
+	This mutex must be held while the AP devices passed through to the KVM
+	guest are plugged/unplugged.
+
+  4. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the operation.
+
+* Operations: Assign or unassign an adapter, domain or control domain of a
+	      mediated device under the control of the vfio_ap device driver:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held while accessing the KVM pointer
+	(matrix_dev->kvm) to plug/unplug AP devices passed through to the
+	KVM guest as a result of the assignment/unassignment operation.
+	Assignment of an AP device may result in additional queue devices
+	or control domains being plugged into the guest. Similarly, unassignment
+	may result in unplugging queue devices or control domains from the
+	guest
+
+  3. KVM Lock (matrix_mdev->kvm->lock):
+
+	This mutex must be held while the AP devices passed through to the KVM
+	guest are plugged in or unplugged.
+
+  4. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the operation. For example, to
+	determine which AP devices need to be plugged/unplugged, the lock
+	must be held to prevent other operations from changing the data used
+	to construct the guest's AP configuration.
+
+* Operations: Probe or remove an AP queue device:
+
+  When a queue device is bound to the vfio_ap device driver, the driver's probe
+  callback is invoked. Similarly, when a queue device is unbound from the
+  driver it's remove callback is invoked. The probe and remove functions will
+  take locks in the following order:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held for the duration of this operation.
+
+	At the time of the operation, the vfio_ap device driver will only have
+	the APQN of the queue being probed or removed, so the
+	matrix_dev->mdevs_list must be perused to locate the matrix_mdev
+	instance to which the queue is assigned. The Guests Lock must be held
+	during this time to prevent the list from being changed while processing
+	the probe/remove.
+
+	Once the matrix_mdev is found, the operation must determine whether the
+	mediated device is passed through to a guest (matrix_mdev->kvm != NULL),
+	then use the KVM pointer to perform the plug/unplug operation. Here
+	again, the lock must be held to prevent other operations from accessing
+	the KVM pointer for the same purpose.
+
+  2. KVM Lock (kvm->lock):
+
+	This mutex must be held while the AP devices passed through to the KVM
+	guest are plugged in or unplugged to prevent other operations from
+	accessing the guest's state while it is in flux.
+
+  3. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the operation, such as the
+	masks used to construct the KVM guest's AP configuration.
+
+* Operations: Probe or remove a mediated device:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held while adding the matrix_mdev to the
+	matrix_dev->mdev_list during the probe operation or when removing it
+	from the list during the remove operation. This is to prevent access by
+	other functions that must traverse the list to find a matrix_mdev for
+	the purpose of plugging/unplugging AP devices passed through to a KVM
+	guest (i.e., probe/remove queue callbacks), while the list is being
+	modified.
+
+  2. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the probe or remove operations
+	such as initializing the hashtable of queue devices
+	(matrix_mdev->qtable.queues) assigned to the matrix_mdev.
+
+* Operation: Handle interception of the PQAP(AQIC) instruction sub-function:
+
+  1. PQAP Hook Lock (kvm->arch.crypto.pqap_hook_rwsem)
+
+	This semaphore must be held in read mode while retrieving the function
+	pointer (kvm->arch.crypto.pqap_hook) and executing the function that
+	handles the interception of the PQAP(AQIC) instruction sub-function by
+	the host.
+
+  2. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the execution of the
+	PQAP(AQIC) instruction sub-function interception handler. For example,
+	the handler must iterate over the matrix_mdev->qtable.queues hashtable
+	to find the vfio_ap_queue object representing the queue for which
+	interrupts are being enabled or disabled.
+
+  Note: It is not necessary to take the Guests Lock (matrix_dev->guests_lock)
+	or the KVM Lock (matrix_mdev->kvm->lock) because the KVM pointer
+	will not be accessed to plug/unplug AP devices passed through to the
+	guest; it will only be used to allocate or free resources for processing
+	interrupts.
+
+* Operation: Handle AP configuration changed notification:
+
+  The vfio_ap device driver registers a callback function to be notified when
+  the AP bus detects that the host's AP configuration has changed. This can
+  occur due to the addition or removal of AP adapters, domains or control
+  domains via an SE or HMC connected to a DPM enabled LPAR. The objective of the
+  handler is to remove the queues no longer accessible via the host in bulk
+  rather than one queue at a time via the driver's queue device remove callback.
+  The locks and the order in which they must be taken by this operation are:
+
+ 1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held for the duration of the operation to:
+
+	* Iterate over the matrix_dev->mdev_list to find each matrix_mdev from
+	  which a queue device to be removed is assigned and prevent other
+	  operations from modifying the list while processing the affected
+	  matrix_mdev instances.
+
+	* Prevent other operations from acquiring access to the KVM pointer in
+	  each affected matrix_mdev instance (matrix_mdev->kvm) for the purpose
+	  of plugging/unplugging AP devices passed through to the KVM guest via
+	  that instance.
+
+2. KVM Lock (kvm->lock):
+
+	This mutex must be held for each affected matrix_mdev instance while
+	the AP devices passed through to the KVM guest are unplugged to prevent
+	other operations from accessing the guest's state while it is in flux.
+
+	Note: This lock must be re-acquired for each matrix_mdev instance.
+
+  3. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held for each affected matrix_mdev to prevent access
+	to the matrix_mdev state while writing/reading state values during the
+	operation, such as the masks used to construct the KVM guest's AP
+	configuration.
+
+	Note: This lock must be re-acquired for each matrix_mdev instance.
+
+Operation: Handle AP bus scan complete notification:
+
+  The vfio_ap device driver registers a callback function to be notified when
+  the AP bus scan completes after detecting the addition or removal of AP
+  adapters, domains or control domains. The objective of the handler is t
+  add the new queues accessible via the host in bulk rather than one queue
+  at a time via the driver's queue device probe callback. The locks and the
+  order in which they must be taken by this operation are:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held for the duration of the operation to:
+
+	* Iterate over the matrix_dev->mdev_list to find each matrix_mdev to
+	  which a queue device added is assigned and prevent other operations
+	  from modifying the list while processing each affected matrix_mdev
+	  object.
+
+	* Prevent other operations from acquiring access to the KVM pointer in
+	  each affected matrix_mdev instance (matrix_mdev->kvm) for the purpose
+	  of plugging/unplugging AP devices passed through to the KVM guest via
+	  that instance.
+
+  2. KVM Lock (kvm->lock):
+
+	This mutex must be held for each affected matrix_mdev instance while
+	the AP devices passed through to the KVM guest are plugged in to prevent
+	other operations from accessing the guest's state while it is in flux.
+
+	Note: This lock must be re-acquired for each matrix_mdev instance.
+
+  3. Matrix Devices Lock (matrix_dev->mdevs_lock):
+
+	This lock must be held for each affected matrix_mdev to prevent access
+	to the matrix_mdev state while writing/reading state values during the
+	operation, such as the masks used to construct the KVM guest's AP
+	configuration.
+
+	Note: This lock must be re-acquired for each matrix_mdev instance.
+
+Operation: Handle resource in use query:
+
+  The vfio_ap device driver registers a callback function with the AP bus to be
+  called when changes to the bus's sysfs /sys/bus/ap/apmask or
+  /sys/bus/ap/aqmask attributes would result in one or more AP queue devices
+  getting unbound from the vfio_ap device driver to verify none of them are in
+  use by the driver (i.e., assigned to a matrix_mdev instance). If this function
+  is called while an adapter or domain is also being assigned to a mediated
+  device, this could result in a deadlock; for example:
+
+  1. A system administrator assigns an adapter to a mediated device under the
+     control of the vfio_ap device driver. The driver will need to first take
+     the matrix_dev->guests_lock to potentially hot plug the adapter into
+     the KVM guest.
+  2. At the same time, a system administrator sets a bit in the sysfs
+     /sys/bus/ap/ap_mask attribute. To complete the operation, the AP bus
+     must:
+     a. Take the ap_perms_mutex lock to update the object storing the values
+        for the /sys/bus/ap/ap_mask attribute.
+     b. Call the vfio_ap device driver's in-use callback to verify that no
+        queues now being reserved for the default zcrypt drivers are
+        in use by the vfio_ap device driver. To do the verification, the in-use
+        callback function takes the matrix_dev->guests_lock, but has to wait
+        because it is already held by the operation in 1 above.
+  3. The vfio_ap device driver calls an AP bus function to verify that the
+     new queues resulting from the assignment of the adapter in step 1 are
+     not reserved for the default zcrypt device driver. This AP bus function
+     tries to take the ap_perms_mutex lock but gets stuck waiting for the
+     it due to step 2a above.
+
+    Consequently, we have the following deadlock situation:
+
+    matrix_dev->guests_lock locked (1)
+    ap_perms_mutex lock locked (2a)
+    Waiting for matrix_dev->lock (2b) which is currently held (1)
+    Waiting for ap_perms_mutex lock (3) which is currently held (2a)
+
+  To prevent the deadlock scenario, the in_use operation will take the
+  required locks using the mutex_trylock() function and if the lock can not be
+  acquired will terminate and return -EBUSY to indicate the driver is busy
+  processing another request.
+
+  The locks required to respond to the handle resource in use query and the
+  order in which they must be taken are:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+  This mutex must be held for the duration of the operation to iterate over the
+  matrix_dev->mdev_list to determine whether any of the queues to be unbound
+  are assigned to a matrix_mdev instance.
+
+  2. Matrix Devices Lock (matrix_dev->mdevs_lock):
+
+  This mutex must be held for the duration of the operation to ensure that the
+  AP configuration of each matrix_mdev instance does not change while verifying
+  that none of the queue devices to be removed from the vfio_ap driver are
+  assigned to it.
author maintainer
#44 lore raw

Re: [PATCH v18 17/18] s390/Docs: new doc describing lock usage by the vfio_ap device driver

Quoted 35 lines · Halil Pasic
Quoted 32 lines · On Mon, 14 Feb 2022 19:50:39 -0500 Tony Krowiak <[email protected]> wrote:

Introduces a new document describing the locks used by the vfio_ap device driver and how to use them so as to avoid lockdep reports and deadlock situations.

---
  Documentation/s390/vfio-ap-locking.rst | 389 +++++++++++++++++++++++++
  1 file changed, 389 insertions(+)

create mode 100644 Documentation/s390/vfio-ap-locking.rst

Documentation/s390/vfio-ap-locking.rst new file +16 -0
new file mode 100644
index 000000000000..10abbb6d6089
@@ -0,0 +1,389 @@
+======================
+VFIO AP Locks Overview
+======================
+This document describes the locks that are pertinent to the secure operation
+of the vfio_ap device driver. Throughout this document, the following variables
+will be used to denote instances of the structures herein described:
+
+struct ap_matrix_dev *matrix_dev;
+struct ap_matrix_mdev *matrix_mdev;
+struct kvm *kvm;
+
+The Matrix Devices Lock (drivers/s390/crypto/vfio_ap_private.h)
+--------------------------------------------------------------
+
+struct ap_matrix_dev {
+	...
Quoted 8 lines

Documentation/s390/vfio-ap-locking.rst line 23

Quoted 7 lines
Documentation/s390/vfio-ap-locking.rst new file +7 -0
+	struct list_head mdev_list;
+	struct mutex mdevs_lock;
+	...
+}
+
+The Matrix Devices Lock (matrix_dev->mdevs_lock) is implemented as a global
+mutex contained within the single instance of struct ap_matrix_dev. This lock

s/single instance of/singleton object of type/

I don't see the problem with instance, but I'll go ahead and make the change.

Quoted 5 lines

Documentation/s390/vfio-ap-locking.rst line 25

Quoted 3 lines
Documentation/s390/vfio-ap-locking.rst new file +2 -0
+controls access to all fields contained within each matrix_mdev instance under
+the control of the vfio_ap device driver (matrix_dev->mdev_list).

Are there matrix_mdev instances not under the control of the vfio_ap device driver?

No, but it doesn't make the statement any less true. I'll take it out, however.

Quoted 9 lines

(MARK 1)

Documentation/s390/vfio-ap-locking.rst line 28

Quoted 4 lines

This lock must

Documentation/s390/vfio-ap-locking.rst new file +3 -0
+be held while reading from, writing to or using the data from a field contained
+within a matrix_mdev instance representing one of the vfio_ap device driver's
+mediated devices.

This makes it look like for example struct vfio_ap_queue objects are out of scope.

How so? The vfio_ap_queue objects are linked to the ap_matrix_mdev object to which the APQN is assigned. Other than that, they are contained in the driver data of the queue device.

Quoted 29 lines
Quoted 29 lines
Documentation/s390/vfio-ap-locking.rst new file +28 -0
+
+The KVM Lock (include/linux/kvm_host.h)
+---------------------------------------
+
+struct kvm {
+	...
+	struct mutex lock;
+	...
+}
+
+The KVM Lock (kvm->lock) controls access to the state data for a KVM guest. This
+lock must be held by the vfio_ap device driver while one or more AP adapters,
+domains or control domains are being plugged into or unplugged from the guest.
+
+The vfio_ap device driver registers a function to be notified when the pointer
+to the kvm instance has been set. The KVM pointer is passed to the handler by
+the notifier and is stored in the in the matrix_mdev instance
+(matrix_mdev->kvm = kvm) containing the state of the mediated device that has
+been passed through to the KVM guest.
+
+The Guests Lock (drivers/s390/crypto/vfio_ap_private.h)
+-----------------------------------------------------------
+
+struct ap_matrix_dev {
+	...
+	struct list_head mdev_list;
+	struct mutex guests_lock;
+	...
Quoted 8 lines

Documentation/s390/vfio-ap-locking.rst line 61

Quoted 5 lines
Documentation/s390/vfio-ap-locking.rst new file +5 -0
+}
+
+The Guests Lock (matrix_dev->guests_lock) controls access to the
+matrix_mdev instances (matrix_dev->mdev_list) that represent mediated devices
+that hold the state for the mediated devices that have been passed through to a

Didn't say that access to fields of matrix_mdev instances is controlled by the matrix_dev->mdevs lock at (MARK 1)? How do the two statements mesh?

The matrix_dev->mdevs_lock controls access to all FIELDS contained within each matrix_mdev and the matrix_dev->guests_lock controls access to matrix_mdev instances; in other words, the matrix_dev->mdev_list and the matrix_mdev instances for the purposes further described below.

Quoted 23 lines
Quoted 23 lines
Documentation/s390/vfio-ap-locking.rst new file +22 -0
+KVM guest. This lock must be held:
+
+1. To control access to the KVM pointer (matrix_mdev->kvm) while the vfio_ap
+   device driver is using it to plug/unplug AP devices passed through to the KVM
+   guest.
+
+2. To add matrix_mdev instances to or remove them from matrix_dev->mdev_list.
+   This is necessary to ensure the proper locking order when the list is perused
+   to find an ap_matrix_mdev instance for the purpose of plugging/unplugging
+   AP devices passed through to a KVM guest.
+
+   For example, when a queue device is removed from the vfio_ap device driver,
+   if the adapter is passed through to a KVM guest, it will have to be
+   unplugged. In order to figure out whether the adapter is passed through,
+   the matrix_mdev object to which the queue is assigned will have to be
+   found. The KVM pointer (matrix_mdev->kvm) can then be used to determine if
+   the mediated device is passed through (matrix_mdev->kvm != NULL) and if so,
+   to unplug the adapter.
+
+It is not necessary to take the Guests Lock to access the KVM pointer if the
+pointer is not used to plug/unplug devices passed through to the KVM guest;
+however, in this case, the Matrix Devices Lock (matrix_dev->mdevs_lock) must be
Quoted 8 lines

Documentation/s390/vfio-ap-locking.rst line 89

Quoted 6 lines
Documentation/s390/vfio-ap-locking.rst new file +6 -0
+held in order to access the KVM pointer since it set and cleared under the
+protection of the Matrix Devices Lock. A case in point is the function that
+handles interception of the PQAP(AQIC) instruction sub-function. This handler
+needs to access the KVM pointer only for the purposes of setting or clearing IRQ
+resources, so only the matrix_dev->mdevs_lock needs to be held.
+

It is very unclear what this lock is actually protecting, and when does it need to be taken.

I don't know how I can make it clearer. In 1 above, it states the it protects access to the KVM pointer when it is being used to plug/unplug AP devices. In other words, if the matrix_mdev->kvm pointer is being accessed just
to verify whether the mdev is attached to a guest or not, it is not necessary to take the matrix_dev->guests_lock. On the other hand, whenever the matrix_mdev->kvm pointer is being taken to dynamically update the guest's APCB (i.e., hot plug/unplug AP devices), the matrix_dev->guests_lock must be held. Maybe if I had said hot plug/unplug it would be clearer? I'm open to suggestions.

Quoted 17 lines
Quoted 17 lines
Documentation/s390/vfio-ap-locking.rst new file +16 -0
+The PQAP Hook Lock (arch/s390/include/asm/kvm_host.h)
+-----------------------------------------------------
+
+typedef int (*crypto_hook)(struct kvm_vcpu *vcpu);
+
+struct kvm_s390_crypto {
+	...
+	struct rw_semaphore pqap_hook_rwsem;
+	crypto_hook *pqap_hook;
+	...
+};
+
+The PQAP Hook Lock is a r/w semaphore that controls access to the function
+pointer of the handler (*kvm->arch.crypto.pqap_hook) to invoke when the
+PQAP(AQIC) instruction sub-function is intercepted by the host. The lock must be
+held in write mode when pqap_hook value is set, and in read mode when the
Quoted 8 lines

Documentation/s390/vfio-ap-locking.rst

Quoted 7 lines
Documentation/s390/vfio-ap-locking.rst new file +7 -0
+pqap_hook function is called.
+
+Locking Order
+-------------
+
+If the various locks are not taken in the proper order, it could potentially
+result in a lockdep splat.

Just in a lockdep splat or in a deadlock?

I've never actually encountered a deadlock condition while testing, only a lockdep splat indicating a deadlock could occur. I'll go ahead and add 'deadlock or lockdep splat'.

Quoted 6 lines
Quoted 3 lines

The proper order for taking locks depends upon
+the operation taking place,

That sounds very fishy! The whole point of having a locking order is preventing deadlocks if everybody sticks to the locking order. If there are exceptions, i.e. if we violate the locking order, we risk deadlocks.

It may sound fishy, but it's a true statement. For example, there are cases where the mdev is not attached to a KVM guest. In that case, the matrix_mdev->kvm pointer will be NULL and the matrix_mdev->kvm->lock will not be taken. Of course, the matrix_dev->guests_lock will still have to be taken before the matrix_dev->mdevs_lock. Maybe I should just make that point clearer here.

After rereading the passage, I think that sentence should be removed. It can be seen in the examples when the KVM lock will not be taken.

Quoted 284 lines

Documentation/s390/vfio-ap-locking.rst line 119

Quoted 8 lines

but in general the Guests Lock

Documentation/s390/vfio-ap-locking.rst new file +6 -0
+(matrix_dev->guests_lock) must be taken outside of the KVM Lock (kvm->lock)
+which in turn must be taken outside of the Matrix Devices Lock
+(matrix_dev->mdevs_lock).
+
+The following describes the various operations under which the various locks are
+taken, the purpose for taking them and the order in which they must be taken.

It looks a little odd that you describe all the operations here, especially in prosa.

But hey the good thing is, none seem to violate the lock order.

Quoted 270 lines
Documentation/s390/vfio-ap-locking.rst new file +270 -0
+
+* Operations: Setting or clearing the KVM pointer (matrix_mdev->kvm):
+
+  1. PQAP Hook Lock (kvm->arch.crypto.pqap_hook_rwsem):
+
+	This semaphore must be held in write mode while setting or clearing the
+	reference to the function pointer (kvm->arch.crypt.pqap_hook) to call
+	when the PQAP(AQIC) instruction sub-function is intercepted by the host.
+	The function pointer is set when the KVM pointer is being set and
+	cleared when the KVM pointer is being cleared.
+
+  2.Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held while accessing the KVM pointer
+	(matrix_mdev->kvm) to plug/unplug AP devices passed through to the
+	KVM guest
+
+  3. KVM Lock (kvm->lock):
+
+	This mutex must be held while the AP devices passed through to the KVM
+	guest are plugged/unplugged.
+
+  4. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the operation.
+
+* Operations: Assign or unassign an adapter, domain or control domain of a
+	      mediated device under the control of the vfio_ap device driver:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held while accessing the KVM pointer
+	(matrix_dev->kvm) to plug/unplug AP devices passed through to the
+	KVM guest as a result of the assignment/unassignment operation.
+	Assignment of an AP device may result in additional queue devices
+	or control domains being plugged into the guest. Similarly, unassignment
+	may result in unplugging queue devices or control domains from the
+	guest
+
+  3. KVM Lock (matrix_mdev->kvm->lock):
+
+	This mutex must be held while the AP devices passed through to the KVM
+	guest are plugged in or unplugged.
+
+  4. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the operation. For example, to
+	determine which AP devices need to be plugged/unplugged, the lock
+	must be held to prevent other operations from changing the data used
+	to construct the guest's AP configuration.
+
+* Operations: Probe or remove an AP queue device:
+
+  When a queue device is bound to the vfio_ap device driver, the driver's probe
+  callback is invoked. Similarly, when a queue device is unbound from the
+  driver it's remove callback is invoked. The probe and remove functions will
+  take locks in the following order:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held for the duration of this operation.
+
+	At the time of the operation, the vfio_ap device driver will only have
+	the APQN of the queue being probed or removed, so the
+	matrix_dev->mdevs_list must be perused to locate the matrix_mdev
+	instance to which the queue is assigned. The Guests Lock must be held
+	during this time to prevent the list from being changed while processing
+	the probe/remove.
+
+	Once the matrix_mdev is found, the operation must determine whether the
+	mediated device is passed through to a guest (matrix_mdev->kvm != NULL),
+	then use the KVM pointer to perform the plug/unplug operation. Here
+	again, the lock must be held to prevent other operations from accessing
+	the KVM pointer for the same purpose.
+
+  2. KVM Lock (kvm->lock):
+
+	This mutex must be held while the AP devices passed through to the KVM
+	guest are plugged in or unplugged to prevent other operations from
+	accessing the guest's state while it is in flux.
+
+  3. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the operation, such as the
+	masks used to construct the KVM guest's AP configuration.
+
+* Operations: Probe or remove a mediated device:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held while adding the matrix_mdev to the
+	matrix_dev->mdev_list during the probe operation or when removing it
+	from the list during the remove operation. This is to prevent access by
+	other functions that must traverse the list to find a matrix_mdev for
+	the purpose of plugging/unplugging AP devices passed through to a KVM
+	guest (i.e., probe/remove queue callbacks), while the list is being
+	modified.
+
+  2. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the probe or remove operations
+	such as initializing the hashtable of queue devices
+	(matrix_mdev->qtable.queues) assigned to the matrix_mdev.
+
+* Operation: Handle interception of the PQAP(AQIC) instruction sub-function:
+
+  1. PQAP Hook Lock (kvm->arch.crypto.pqap_hook_rwsem)
+
+	This semaphore must be held in read mode while retrieving the function
+	pointer (kvm->arch.crypto.pqap_hook) and executing the function that
+	handles the interception of the PQAP(AQIC) instruction sub-function by
+	the host.
+
+  2. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held to prevent access to the matrix_mdev state
+	while writing/reading state values during the execution of the
+	PQAP(AQIC) instruction sub-function interception handler. For example,
+	the handler must iterate over the matrix_mdev->qtable.queues hashtable
+	to find the vfio_ap_queue object representing the queue for which
+	interrupts are being enabled or disabled.
+
+  Note: It is not necessary to take the Guests Lock (matrix_dev->guests_lock)
+	or the KVM Lock (matrix_mdev->kvm->lock) because the KVM pointer
+	will not be accessed to plug/unplug AP devices passed through to the
+	guest; it will only be used to allocate or free resources for processing
+	interrupts.
+
+* Operation: Handle AP configuration changed notification:
+
+  The vfio_ap device driver registers a callback function to be notified when
+  the AP bus detects that the host's AP configuration has changed. This can
+  occur due to the addition or removal of AP adapters, domains or control
+  domains via an SE or HMC connected to a DPM enabled LPAR. The objective of the
+  handler is to remove the queues no longer accessible via the host in bulk
+  rather than one queue at a time via the driver's queue device remove callback.
+  The locks and the order in which they must be taken by this operation are:
+
+ 1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held for the duration of the operation to:
+
+	* Iterate over the matrix_dev->mdev_list to find each matrix_mdev from
+	  which a queue device to be removed is assigned and prevent other
+	  operations from modifying the list while processing the affected
+	  matrix_mdev instances.
+
+	* Prevent other operations from acquiring access to the KVM pointer in
+	  each affected matrix_mdev instance (matrix_mdev->kvm) for the purpose
+	  of plugging/unplugging AP devices passed through to the KVM guest via
+	  that instance.
+
+2. KVM Lock (kvm->lock):
+
+	This mutex must be held for each affected matrix_mdev instance while
+	the AP devices passed through to the KVM guest are unplugged to prevent
+	other operations from accessing the guest's state while it is in flux.
+
+	Note: This lock must be re-acquired for each matrix_mdev instance.
+
+  3. Matrix Devices Lock (matrix_dev->mdevs_lock)
+
+	This lock must be held for each affected matrix_mdev to prevent access
+	to the matrix_mdev state while writing/reading state values during the
+	operation, such as the masks used to construct the KVM guest's AP
+	configuration.
+
+	Note: This lock must be re-acquired for each matrix_mdev instance.
+
+Operation: Handle AP bus scan complete notification:
+
+  The vfio_ap device driver registers a callback function to be notified when
+  the AP bus scan completes after detecting the addition or removal of AP
+  adapters, domains or control domains. The objective of the handler is t
+  add the new queues accessible via the host in bulk rather than one queue
+  at a time via the driver's queue device probe callback. The locks and the
+  order in which they must be taken by this operation are:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+	This mutex must be held for the duration of the operation to:
+
+	* Iterate over the matrix_dev->mdev_list to find each matrix_mdev to
+	  which a queue device added is assigned and prevent other operations
+	  from modifying the list while processing each affected matrix_mdev
+	  object.
+
+	* Prevent other operations from acquiring access to the KVM pointer in
+	  each affected matrix_mdev instance (matrix_mdev->kvm) for the purpose
+	  of plugging/unplugging AP devices passed through to the KVM guest via
+	  that instance.
+
+  2. KVM Lock (kvm->lock):
+
+	This mutex must be held for each affected matrix_mdev instance while
+	the AP devices passed through to the KVM guest are plugged in to prevent
+	other operations from accessing the guest's state while it is in flux.
+
+	Note: This lock must be re-acquired for each matrix_mdev instance.
+
+  3. Matrix Devices Lock (matrix_dev->mdevs_lock):
+
+	This lock must be held for each affected matrix_mdev to prevent access
+	to the matrix_mdev state while writing/reading state values during the
+	operation, such as the masks used to construct the KVM guest's AP
+	configuration.
+
+	Note: This lock must be re-acquired for each matrix_mdev instance.
+
+Operation: Handle resource in use query:
+
+  The vfio_ap device driver registers a callback function with the AP bus to be
+  called when changes to the bus's sysfs /sys/bus/ap/apmask or
+  /sys/bus/ap/aqmask attributes would result in one or more AP queue devices
+  getting unbound from the vfio_ap device driver to verify none of them are in
+  use by the driver (i.e., assigned to a matrix_mdev instance). If this function
+  is called while an adapter or domain is also being assigned to a mediated
+  device, this could result in a deadlock; for example:
+
+  1. A system administrator assigns an adapter to a mediated device under the
+     control of the vfio_ap device driver. The driver will need to first take
+     the matrix_dev->guests_lock to potentially hot plug the adapter into
+     the KVM guest.
+  2. At the same time, a system administrator sets a bit in the sysfs
+     /sys/bus/ap/ap_mask attribute. To complete the operation, the AP bus
+     must:
+     a. Take the ap_perms_mutex lock to update the object storing the values
+        for the /sys/bus/ap/ap_mask attribute.
+     b. Call the vfio_ap device driver's in-use callback to verify that no
+        queues now being reserved for the default zcrypt drivers are
+        in use by the vfio_ap device driver. To do the verification, the in-use
+        callback function takes the matrix_dev->guests_lock, but has to wait
+        because it is already held by the operation in 1 above.
+  3. The vfio_ap device driver calls an AP bus function to verify that the
+     new queues resulting from the assignment of the adapter in step 1 are
+     not reserved for the default zcrypt device driver. This AP bus function
+     tries to take the ap_perms_mutex lock but gets stuck waiting for the
+     it due to step 2a above.
+
+    Consequently, we have the following deadlock situation:
+
+    matrix_dev->guests_lock locked (1)
+    ap_perms_mutex lock locked (2a)
+    Waiting for matrix_dev->lock (2b) which is currently held (1)
+    Waiting for ap_perms_mutex lock (3) which is currently held (2a)
+
+  To prevent the deadlock scenario, the in_use operation will take the
+  required locks using the mutex_trylock() function and if the lock can not be
+  acquired will terminate and return -EBUSY to indicate the driver is busy
+  processing another request.
+
+  The locks required to respond to the handle resource in use query and the
+  order in which they must be taken are:
+
+  1. Guests Lock (matrix_dev->guests_lock):
+
+  This mutex must be held for the duration of the operation to iterate over the
+  matrix_dev->mdev_list to determine whether any of the queues to be unbound
+  are assigned to a matrix_mdev instance.
+
+  2. Matrix Devices Lock (matrix_dev->mdevs_lock):
+
+  This mutex must be held for the duration of the operation to ensure that the
+  AP configuration of each matrix_mdev instance does not change while verifying
+  that none of the queue devices to be removed from the vfio_ap driver are
+  assigned to it.
maintainer
#45 lore raw

Re: [PATCH v18 17/18] s390/Docs: new doc describing lock usage by the vfio_ap device driver

Quoted 45 lines · On Mon, 4 Apr 2022 17:34:48 -0400 Tony Krowiak <[email protected]> wrote:
Quoted 35 lines · Halil Pasic
Quoted 32 lines · On Mon, 14 Feb 2022 19:50:39 -0500 Tony Krowiak <[email protected]> wrote:

Introduces a new document describing the locks used by the vfio_ap device driver and how to use them so as to avoid lockdep reports and deadlock situations.

---
  Documentation/s390/vfio-ap-locking.rst | 389 +++++++++++++++++++++++++
  1 file changed, 389 insertions(+)

create mode 100644 Documentation/s390/vfio-ap-locking.rst

Documentation/s390/vfio-ap-locking.rst new file +16 -0
new file mode 100644
index 000000000000..10abbb6d6089
@@ -0,0 +1,389 @@
+======================
+VFIO AP Locks Overview
+======================
+This document describes the locks that are pertinent to the secure operation
+of the vfio_ap device driver. Throughout this document, the following variables
+will be used to denote instances of the structures herein described:
+
+struct ap_matrix_dev *matrix_dev;
+struct ap_matrix_mdev *matrix_mdev;
+struct kvm *kvm;
+
+The Matrix Devices Lock (drivers/s390/crypto/vfio_ap_private.h)
+--------------------------------------------------------------
+
+struct ap_matrix_dev {
+	...
Quoted 8 lines

Documentation/s390/vfio-ap-locking.rst line 23

Quoted 7 lines
Documentation/s390/vfio-ap-locking.rst new file +7 -0
+	struct list_head mdev_list;
+	struct mutex mdevs_lock;
+	...
+}
+
+The Matrix Devices Lock (matrix_dev->mdevs_lock) is implemented as a global
+mutex contained within the single instance of struct ap_matrix_dev. This lock  

s/single instance of/singleton object of type/

Quoted 2 lines

I don't see the problem with instance, but I'll go ahead and make the change.

My problem is not with the word "instance", and problem is too strong word anyway. I think that "the single instance of" is a little vague, because there is ambiguity about the singularity (and existence) of the single instance. If you have multiple servers, you may have several "the single instances" in that system (one per kernel, or none if no vfio-ap module is loaded for example). If you have a nested visualization setup, one can even argue that there are multiple instances of "the single instance" within one Linux system.

I see an advantage in using the "singleton object", because most of us have at least heard of the singleton design pattern, if not learned about it in class, and thus the scope of singularity is much better defined.

Quoted 8 lines
Quoted 6 lines

Documentation/s390/vfio-ap-locking.rst line 25

Quoted 4 lines
Documentation/s390/vfio-ap-locking.rst new file +2 -0
+controls access to all fields contained within each matrix_mdev instance under
+the control of the vfio_ap device driver (matrix_dev->mdev_list).  

Are there matrix_mdev instances not under the control of the vfio_ap device driver?

No, but it doesn't make the statement any less true.

No it does not make it less true, just more confusing. By that logic you could logical and every of your statements with all the tautologies of this world.

Quoted 6 lines

I'll take it out,
however.

Quoted 3 lines

(MARK 1)

Quoted 8 lines
Quoted 6 lines

Documentation/s390/vfio-ap-locking.rst line 28

Quoted 4 lines

This lock must

Documentation/s390/vfio-ap-locking.rst new file +3 -0
+be held while reading from, writing to or using the data from a field contained
+within a matrix_mdev instance representing one of the vfio_ap device driver's
+mediated devices.  

This makes it look like for example struct vfio_ap_queue objects are out of scope.

How so? The vfio_ap_queue objects are linked to the ap_matrix_mdev object

The *key* are the words "linked" versus "a field contained within a matrix_mdev".

Quoted 3 lines

to which the APQN is assigned. Other than that, they are contained in the driver
data of the queue device.

The vfio_ap_queue object a separate object allocated in vfio_ap_mdev_probe_queue() and is certainly not a field contained within a matrix_mdev.

Please notice that if you were to extend to all the objects reachable from matrix_mdev instances, you would be in trouble, because a pointer to kvm is also reachable, and via that pointer an awful lot of things that are certainly out of scope.

Quoted 39 lines
Quoted 30 lines
Quoted 30 lines
Documentation/s390/vfio-ap-locking.rst new file +28 -0
+
+The KVM Lock (include/linux/kvm_host.h)
+---------------------------------------
+
+struct kvm {
+	...
+	struct mutex lock;
+	...
+}
+
+The KVM Lock (kvm->lock) controls access to the state data for a KVM guest. This
+lock must be held by the vfio_ap device driver while one or more AP adapters,
+domains or control domains are being plugged into or unplugged from the guest.
+
+The vfio_ap device driver registers a function to be notified when the pointer
+to the kvm instance has been set. The KVM pointer is passed to the handler by
+the notifier and is stored in the in the matrix_mdev instance
+(matrix_mdev->kvm = kvm) containing the state of the mediated device that has
+been passed through to the KVM guest.
+
+The Guests Lock (drivers/s390/crypto/vfio_ap_private.h)
+-----------------------------------------------------------
+
+struct ap_matrix_dev {
+	...
+	struct list_head mdev_list;
+	struct mutex guests_lock;
+	...
Quoted 8 lines

Documentation/s390/vfio-ap-locking.rst line 61

Quoted 5 lines
Documentation/s390/vfio-ap-locking.rst new file +5 -0
+}
+
+The Guests Lock (matrix_dev->guests_lock) controls access to the
+matrix_mdev instances (matrix_dev->mdev_list) that represent mediated devices
+that hold the state for the mediated devices that have been passed through to a  

Didn't say that access to fields of matrix_mdev instances is controlled by the matrix_dev->mdevs lock at (MARK 1)? How do the two statements mesh?

Quoted 5 lines

The matrix_dev->mdevs_lock controls access to all FIELDS contained within each matrix_mdev and the matrix_dev->guests_lock controls access to matrix_mdev instances; in other words, the matrix_dev->mdev_list and the matrix_mdev instances for the purposes further described below.

See above.

Quoted 74 lines
Quoted 25 lines
Quoted 25 lines
Documentation/s390/vfio-ap-locking.rst new file +22 -0
+KVM guest. This lock must be held:
+
+1. To control access to the KVM pointer (matrix_mdev->kvm) while the vfio_ap
+   device driver is using it to plug/unplug AP devices passed through to the KVM
+   guest.
+
+2. To add matrix_mdev instances to or remove them from matrix_dev->mdev_list.
+   This is necessary to ensure the proper locking order when the list is perused
+   to find an ap_matrix_mdev instance for the purpose of plugging/unplugging
+   AP devices passed through to a KVM guest.
+
+   For example, when a queue device is removed from the vfio_ap device driver,
+   if the adapter is passed through to a KVM guest, it will have to be
+   unplugged. In order to figure out whether the adapter is passed through,
+   the matrix_mdev object to which the queue is assigned will have to be
+   found. The KVM pointer (matrix_mdev->kvm) can then be used to determine if
+   the mediated device is passed through (matrix_mdev->kvm != NULL) and if so,
+   to unplug the adapter.
+
+It is not necessary to take the Guests Lock to access the KVM pointer if the
+pointer is not used to plug/unplug devices passed through to the KVM guest;
+however, in this case, the Matrix Devices Lock (matrix_dev->mdevs_lock) must be
Quoted 8 lines

Documentation/s390/vfio-ap-locking.rst line 89

Quoted 6 lines
Documentation/s390/vfio-ap-locking.rst new file +6 -0
+held in order to access the KVM pointer since it set and cleared under the
+protection of the Matrix Devices Lock. A case in point is the function that
+handles interception of the PQAP(AQIC) instruction sub-function. This handler
+needs to access the KVM pointer only for the purposes of setting or clearing IRQ
+resources, so only the matrix_dev->mdevs_lock needs to be held.
+  

It is very unclear what this lock is actually protecting, and when does it need to be taken.

I don't know how I can make it clearer. In 1 above, it states the it protects
access to the KVM pointer when it is being used to plug/unplug AP devices. In other words, if the matrix_mdev->kvm pointer is being accessed just to verify whether the mdev is attached to a guest or not, it is not necessary to
take the matrix_dev->guests_lock. On the other hand, whenever the matrix_mdev->kvm
pointer is being taken to dynamically update the guest's APCB (i.e., hot plug/unplug AP
devices), the matrix_dev->guests_lock must be held. Maybe if I had said hot plug/unplug
it would be clearer? I'm open to suggestions.

Quoted 17 lines
Quoted 17 lines
Documentation/s390/vfio-ap-locking.rst new file +16 -0
+The PQAP Hook Lock (arch/s390/include/asm/kvm_host.h)
+-----------------------------------------------------
+
+typedef int (*crypto_hook)(struct kvm_vcpu *vcpu);
+
+struct kvm_s390_crypto {
+	...
+	struct rw_semaphore pqap_hook_rwsem;
+	crypto_hook *pqap_hook;
+	...
+};
+
+The PQAP Hook Lock is a r/w semaphore that controls access to the function
+pointer of the handler (*kvm->arch.crypto.pqap_hook) to invoke when the
+PQAP(AQIC) instruction sub-function is intercepted by the host. The lock must be
+held in write mode when pqap_hook value is set, and in read mode when the
Quoted 8 lines

Documentation/s390/vfio-ap-locking.rst

Quoted 7 lines
Documentation/s390/vfio-ap-locking.rst new file +7 -0
+pqap_hook function is called.
+
+Locking Order
+-------------
+
+If the various locks are not taken in the proper order, it could potentially
+result in a lockdep splat.  

Just in a lockdep splat or in a deadlock?

Quoted 3 lines

I've never actually encountered a deadlock condition while testing, only a lockdep splat indicating a deadlock could occur. I'll go ahead and add 'deadlock or lockdep splat'.

Sorry, it is my sensitivity regarding situations were people claim they are just fixing a compiler warning, where in reality that compiler warning is just drawing attention to a severe bug. In my eyes lockdep is just a tool to detect locking problems. Just focusing on the tool is missing the point.

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

The proper order for taking locks depends upon
+the operation taking place,

That sounds very fishy! The whole point of having a locking order is preventing deadlocks if everybody sticks to the locking order. If there are exceptions, i.e. if we violate the locking order, we risk deadlocks.

Quoted 6 lines

It may sound fishy, but it's a true statement. For example, there are cases where the mdev is not attached to a KVM guest. In that case, the matrix_mdev->kvm pointer will be NULL and the matrix_mdev->kvm->lock will not be taken. Of course, the matrix_dev->guests_lock will still have to be taken before the matrix_dev->mdevs_lock.

But the locks are still taken in the very same order. You may never have a situation where you first blocking-take A with B held, and another one where you blocking-take B with A held. That is what "locking order" is about in CS.

Quoted 7 lines

Maybe I should just make that point
clearer here.

After rereading the passage, I think that sentence should be removed. It can be seen in the examples when the KVM lock will not be taken.

No having it on one place is great. If you don't state the lock hierarchy in one place, people would need to extract it from the jungle of scenarios.

[..]

Regards,
Halil