Skip to contentSkip to conversation

vfio-ccw: Enable transparent CCW IPL from DASD

PATCH Superseded Jared Rossi 3 participants 4 messages May 6, 2020

Subsystems (5)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover vfio-ccw: Enable transparent CCW IPL from DASD

Remove the explicit prefetch check when using vfio-ccw devices. This check does not trigger in practice as all Linux channel programs are intended to use prefetch.

Version 3 improves logging by including the UUID of the vfio device that triggers the warning. A custom rate limit is used because the generic rate limit of 10 per 5 seconds will still result in multiple warnings during IPL. The warning message has been clarfied to reflect that a channel program will be executed using prefetch even though prefetch was not specified.

The text of warning itself does not explicitly refer to non-prefetching channel programs as unsupported because it will trigger during IPL, which is a normal and expected sequence. Likewise, because we expect the message to appear during IPL, the warning also does not explicitly alert to the potential of an error, rather it simply notes that a channel program is being executed in a way other than specified.

Verson 3 also makes some word choice changes to the documentation.

Jared Rossi (1):
vfio-ccw: Enable transparent CCW IPL from DASD

 Documentation/s390/vfio-ccw.rst |  6 ++++++
 drivers/s390/cio/vfio_ccw_cp.c  | 19 ++++++++++++-------
 2 files changed, 18 insertions(+), 7 deletions(-)
Signature · 1 line
2.17.0
#1 lore raw

PATCH v3 1/1 vfio-ccw: Enable transparent CCW IPL from DASD

Remove the explicit prefetch check when using vfio-ccw devices. This check does not trigger in practice as all Linux channel programs are intended to use prefetch.

It is expected that all ORBs issued by Linux will request prefetch. Although non-prefetching ORBs are not rejected, they will prefetch nonetheless. A warning is issued up to once per 5 seconds when a forced prefetch occurs.

A non-prefetch ORB does not necessarily result in an error, however frequent encounters with non-prefetch ORBs indicate that channel programs are being executed in a way that is inconsistent with what the guest is requesting. While there is currently no known case of an error caused by forced prefetch, it is possible in theory that forced prefetch could result in an error if applied to a channel program that is dependent on non-prefetch.

---
 Documentation/s390/vfio-ccw.rst |  6 ++++++
 drivers/s390/cio/vfio_ccw_cp.c  | 19 ++++++++++++-------
 2 files changed, 18 insertions(+), 7 deletions(-)
Documentation/s390/vfio-ccw.rst +6 -0
index fca9c4f5bd9c..23e7d136f8b4 100644
@@ -335,6 +335,12 @@ device.
 The current code allows the guest to start channel programs via
 START SUBCHANNEL, and to issue HALT SUBCHANNEL and CLEAR SUBCHANNEL.
 
+Currently all channel programs are prefetched, regardless of the
+p-bit setting in the ORB.  As a result, self modifying channel
+programs are not supported.  For this reason, IPL has to be handled as
+a special case by a userspace/guest program; this has been implemented
+in QEMU's s390-ccw bios as of QEMU 4.1.
+
 vfio-ccw supports classic (command mode) channel I/O only. Transport
 mode (HPF) is not supported.
 
drivers/s390/cio/vfio_ccw_cp.c +12 -7
index 3645d1720c4b..d423ca934779 100644
@@ -8,6 +8,7 @@
  *            Xiao Feng Ren <[email protected]>
  */
 
+#include <linux/ratelimit.h>
 #include <linux/mm.h>
 #include <linux/slab.h>
 #include <linux/iommu.h>
@@ -625,23 +626,27 @@ static int ccwchain_fetch_one(struct ccwchain *chain,
  * the target channel program from @orb->cmd.iova to the new ccwchain(s).
  *
  * Limitations:
- * 1. Supports only prefetch enabled mode.
- * 2. Supports idal(c64) ccw chaining.
- * 3. Supports 4k idaw.
+ * 1. Supports idal(c64) ccw chaining.
+ * 2. Supports 4k idaw.
  *
  * Returns:
  *   %0 on success and a negative error value on failure.
  */
 int cp_init(struct channel_program *cp, struct device *mdev, union orb *orb)
 {
+	static DEFINE_RATELIMIT_STATE(ratelimit_state, 5 * HZ, 1);
 	int ret;
 
 	/*
-	 * XXX:
-	 * Only support prefetch enable mode now.
+	 * We only support prefetching the channel program. We assume all channel
+	 * programs executed by supported guests (i.e. Linux) likewise support
+	 * prefetching. Even if prefetching is not specified the channel program
+	 * is still executed using prefetch. Executing a channel program that
+	 * does not specify prefetching will typically not cause an error, but a
+	 * warning is issued to help identify the problem if something does break.
 	 */
-	if (!orb->cmd.pfch)
-		return -EOPNOTSUPP;
+	if (!orb->cmd.pfch && __ratelimit(&ratelimit_state))
+		dev_warn(mdev, "executing channel program with prefetch, but prefetch isn't specified");
 
 	INIT_LIST_HEAD(&cp->ccwchain_list);
 	memcpy(&cp->orb, orb, sizeof(*orb));
Signature · 1 line
2.17.0
#2 lore raw
Quoted 23 lines · On Tue, 5 May 2020 20:15:44 -0400 Jared Rossi <[email protected]> wrote:

Remove the explicit prefetch check when using vfio-ccw devices. This check does not trigger in practice as all Linux channel programs are intended to use prefetch.

It is expected that all ORBs issued by Linux will request prefetch. Although non-prefetching ORBs are not rejected, they will prefetch nonetheless. A warning is issued up to once per 5 seconds when a forced prefetch occurs.

A non-prefetch ORB does not necessarily result in an error, however frequent encounters with non-prefetch ORBs indicate that channel programs are being executed in a way that is inconsistent with what the guest is requesting. While there is currently no known case of an error caused by forced prefetch, it is possible in theory that forced prefetch could result in an error if applied to a channel program that is dependent on non-prefetch.

---
 Documentation/s390/vfio-ccw.rst |  6 ++++++
 drivers/s390/cio/vfio_ccw_cp.c  | 19 ++++++++++++-------
 2 files changed, 18 insertions(+), 7 deletions(-)

(...)

Quoted 20 lines
drivers/s390/cio/vfio_ccw_cp.c +3 -4
@@ -625,23 +626,27 @@ static int ccwchain_fetch_one(struct ccwchain *chain,
  * the target channel program from @orb->cmd.iova to the new ccwchain(s).
  *
  * Limitations:
- * 1. Supports only prefetch enabled mode.
- * 2. Supports idal(c64) ccw chaining.
- * 3. Supports 4k idaw.
+ * 1. Supports idal(c64) ccw chaining.
+ * 2. Supports 4k idaw.
  *
  * Returns:
  *   %0 on success and a negative error value on failure.
  */
 int cp_init(struct channel_program *cp, struct device *mdev, union orb *orb)
 {
+	static DEFINE_RATELIMIT_STATE(ratelimit_state, 5 * HZ, 1);
 	int ret;
 
 	/*
-	 * XXX:

drivers/s390/cio/vfio_ccw_cp.c static int ccwchain_fetch_one(struct ccwchain *chain, · line 644

Quoted 10 lines
drivers/s390/cio/vfio_ccw_cp.c +6 -3
-	 * Only support prefetch enable mode now.
+	 * We only support prefetching the channel program. We assume all channel
+	 * programs executed by supported guests (i.e. Linux) likewise support
+	 * prefetching. Even if prefetching is not specified the channel program
+	 * is still executed using prefetch. Executing a channel program that
+	 * does not specify prefetching will typically not cause an error, but a
+	 * warning is issued to help identify the problem if something does break.
 	 */
-	if (!orb->cmd.pfch)
-		return -EOPNOTSUPP;

/* custom ratelimiting to avoid flood during boot */

(to avoid people stumbling over this)

drivers/s390/cio/vfio_ccw_cp.c static int ccwchain_fetch_one(struct ccwchain *chain, · line 649

Quoted 2 lines
drivers/s390/cio/vfio_ccw_cp.c +2 -0
+	if (!orb->cmd.pfch && __ratelimit(&ratelimit_state))
+		dev_warn(mdev, "executing channel program with prefetch, but prefetch isn't specified");

hmm... what about

"prefetching channel program even though prefetch not specified in orb"?

drivers/s390/cio/vfio_ccw_cp.c static int ccwchain_fetch_one(struct ccwchain *chain, · line 652

Quoted 3 lines
drivers/s390/cio/vfio_ccw_cp.c +0 -0
 	INIT_LIST_HEAD(&cp->ccwchain_list);
 	memcpy(&cp->orb, orb, sizeof(*orb));

(...)

Apart from the bikeshedding, looks sane to me; but would like more opinions.

maintainer
#3 lore raw
In reply to Jared Rossi
Quoted 58 lines · Jared Rossi

Remove the explicit prefetch check when using vfio-ccw devices. This check does not trigger in practice as all Linux channel programs are intended to use prefetch.

It is expected that all ORBs issued by Linux will request prefetch. Although non-prefetching ORBs are not rejected, they will prefetch nonetheless. A warning is issued up to once per 5 seconds when a forced prefetch occurs.

A non-prefetch ORB does not necessarily result in an error, however frequent encounters with non-prefetch ORBs indicate that channel programs are being executed in a way that is inconsistent with what the guest is requesting. While there is currently no known case of an error caused by forced prefetch, it is possible in theory that forced prefetch could result in an error if applied to a channel program that is dependent on non-prefetch.

---
 Documentation/s390/vfio-ccw.rst |  6 ++++++
 drivers/s390/cio/vfio_ccw_cp.c  | 19 ++++++++++++-------
 2 files changed, 18 insertions(+), 7 deletions(-)
Documentation/s390/vfio-ccw.rst +6 -0
index fca9c4f5bd9c..23e7d136f8b4 100644
@@ -335,6 +335,12 @@ device.
 The current code allows the guest to start channel programs via
 START SUBCHANNEL, and to issue HALT SUBCHANNEL and CLEAR SUBCHANNEL.
 
+Currently all channel programs are prefetched, regardless of the
+p-bit setting in the ORB.  As a result, self modifying channel
+programs are not supported.  For this reason, IPL has to be handled as
+a special case by a userspace/guest program; this has been implemented
+in QEMU's s390-ccw bios as of QEMU 4.1.
+
 vfio-ccw supports classic (command mode) channel I/O only. Transport
 mode (HPF) is not supported.
 
drivers/s390/cio/vfio_ccw_cp.c +1 -2
index 3645d1720c4b..d423ca934779 100644
@@ -8,6 +8,7 @@
  *            Xiao Feng Ren <[email protected]>
  */
 
+#include <linux/ratelimit.h>
 #include <linux/mm.h>
 #include <linux/slab.h>
 #include <linux/iommu.h>
@@ -625,23 +626,27 @@ static int ccwchain_fetch_one(struct ccwchain *chain,
  * the target channel program from @orb->cmd.iova to the new ccwchain(s).
  *
  * Limitations:
- * 1. Supports only prefetch enabled mode.
- * 2. Supports idal(c64) ccw chaining.

drivers/s390/cio/vfio_ccw_cp.c static int ccwchain_fetch_one(struct ccwchain *chain, · line 637

Quoted 10 lines
drivers/s390/cio/vfio_ccw_cp.c +3 -1
- * 3. Supports 4k idaw.
+ * 1. Supports idal(c64) ccw chaining.
+ * 2. Supports 4k idaw.
  *
  * Returns:
  *   %0 on success and a negative error value on failure.
  */
 int cp_init(struct channel_program *cp, struct device *mdev, union orb *orb)
 {
+	static DEFINE_RATELIMIT_STATE(ratelimit_state, 5 * HZ, 1);

This looks peculiar, being static and on the stack. But I guess it is fine. And as Conny mentions, a comment about the boot messages would be good.

drivers/s390/cio/vfio_ccw_cp.c static int ccwchain_fetch_one(struct ccwchain *chain, · line 642

Quoted 7 lines
drivers/s390/cio/vfio_ccw_cp.c +2 -2
 	int ret;
 
 	/*
-	 * XXX:
-	 * Only support prefetch enable mode now.
+	 * We only support prefetching the channel program. We assume all channel
+	 * programs executed by supported guests (i.e. Linux) likewise support

s/(i.e. Linux) //

drivers/s390/cio/vfio_ccw_cp.c static int ccwchain_fetch_one(struct ccwchain *chain, · line 643

Quoted 1 line
drivers/s390/cio/vfio_ccw_cp.c +1 -0
+	 * prefetching. 

Even if prefetching is not specified the channel program

drivers/s390/cio/vfio_ccw_cp.c static int ccwchain_fetch_one(struct ccwchain *chain, · line 644

Quoted 1 line
drivers/s390/cio/vfio_ccw_cp.c +1 -0
+	 * is still executed using prefetch. 

The above sentence seems redundant, and can be removed.

Executing a channel program that

drivers/s390/cio/vfio_ccw_cp.c static int ccwchain_fetch_one(struct ccwchain *chain, · line 649

Quoted 7 lines
drivers/s390/cio/vfio_ccw_cp.c +4 -2
+	 * does not specify prefetching will typically not cause an error, but a
+	 * warning is issued to help identify the problem if something does break.
 	 */
-	if (!orb->cmd.pfch)
-		return -EOPNOTSUPP;
+	if (!orb->cmd.pfch && __ratelimit(&ratelimit_state))
+		dev_warn(mdev, "executing channel program with prefetch, but prefetch isn't specified");

Works well enough (with QEMU patch, obviously).

Almost-r-b: me :)

Quoted 4 lines
drivers/s390/cio/vfio_ccw_cp.c +0 -0
 	INIT_LIST_HEAD(&cp->ccwchain_list);
 	memcpy(&cp->orb, orb, sizeof(*orb));