vfio-ccw: Fix interrupt handling for HALT/CLEAR

- Detach the cp from the subchannel (or better, remove the 1:1 relationship). By that I mean building the cp as a separately allocated structure (maybe embedding a kref, but that might not be needed), and appending it to a list after SSCH with cc=0. Discard it if cc!=0.
- Remove the CP_PENDING state. The state is either IDLE after any successful SSCH/HSCH/CSCH, or a new state in that case. But no special state for SSCH.
- A successful CSCH removes the first queued request, if any.
- A final interrupt removes the first queued request, if any.

The interrupt path checks the FSM state when processing a final interrupt (an interrupt that is neither subchannel active, nor device active), to determine whether to call cp_free() and release the associated memory. But, this does not fully close the window where a START comes in after a HALT/CLEAR. If the START runs while the CLEAR interrupt is being processed, the channel program struct will be allocated while the interrupt would be considering whether or not to free it. If the FSM state is CP_PROCESSING, then everything is fine. But if the START is able to issue its SSCH and get a cc0, then the in-flight interrupt would have been for an unrelated operation (perhaps none, if the subchannel was previously idle).

The channel_program struct has an "initialized" flag that is set early in the fsm_io_request() flow, to simplify the various cp_*() accessors. Let's extend this idea to include a "started" flag that announces that the channel program has successfully been issued to hardware. With this, the interrupt path can determine whether the final interrupt should also release the cp resources instead of relying on a transient FSM state.

Signed-off-by Eric Farman <[email protected]>

---
 drivers/s390/cio/vfio_ccw_cp.c  |  2 ++
 drivers/s390/cio/vfio_ccw_cp.h  |  1 +
 drivers/s390/cio/vfio_ccw_drv.c |  2 +-
 drivers/s390/cio/vfio_ccw_fsm.c | 11 +++++++++++
 4 files changed, 15 insertions(+), 1 deletion(-)

drivers/s390/cio/vfio_ccw_cp.c +2 -0

index b9febc581b1f..7748eeef434e 100644
@@ -657,6 +657,7 @@ int cp_init(struct channel_program *cp, struct device *mdev, union orb *orb)
 
 	if (!ret) {
 		cp->initialized = true;
+		cp->started = false;
 
 		/* It is safe to force: if it was not set but idals used
 		 * ccwchain_calc_length would have returned an error.

@@ -685,6 +686,7 @@ void cp_free(struct channel_program *cp)
 		return;
 
 	cp->initialized = false;
+	cp->started = false;
 	list_for_each_entry_safe(chain, temp, &cp->ccwchain_list, next) {
 		for (i = 0; i < chain->ch_len; i++) {
 			pfn_array_unpin_free(chain->ch_pa + i, cp->mdev);

drivers/s390/cio/vfio_ccw_cp.h +1 -0

index ba31240ce965..7ea14910aaaa 100644
@@ -39,6 +39,7 @@ struct channel_program {
 	union orb orb;
 	struct device *mdev;
 	bool initialized;
+	bool started;
 	struct ccw1 *guest_cp;
 };
 

drivers/s390/cio/vfio_ccw_drv.c +1 -1

index 8c625b530035..7e2a790dc9a1 100644
@@ -94,7 +94,7 @@ static void vfio_ccw_sch_io_todo(struct work_struct *work)
 		     (SCSW_ACTL_DEVACT | SCSW_ACTL_SCHACT));
 	if (scsw_is_solicited(&irb->scsw)) {
 		cp_update_scsw(&private->cp, &irb->scsw);
-		if (is_final && private->state == VFIO_CCW_STATE_CP_PENDING)
+		if (is_final && private->cp.started)
 			cp_free(&private->cp);
 	}
 	mutex_lock(&private->io_mutex);

drivers/s390/cio/vfio_ccw_fsm.c +11 -0

index 23e61aa638e4..d806f88eba72 100644
@@ -50,6 +50,7 @@ static int fsm_io_helper(struct vfio_ccw_private *private)
 		sch->schib.scsw.cmd.actl |= SCSW_ACTL_START_PEND;
 		ret = 0;
 		private->state = VFIO_CCW_STATE_CP_PENDING;
+		private->cp.started = true;
 		break;
 	case 1:		/* Status pending */
 	case 2:		/* Busy */

@@ -246,6 +247,16 @@ static void fsm_io_request(struct vfio_ccw_private *private,
 	char *errstr = "request";
 	struct subchannel_id schid = get_schid(private);
 
+	if (private->cp.started) {
+		io_region->ret_code = -EBUSY;
+		VFIO_CCW_MSG_EVENT(2,
+				   "%pUl (%x.%x.%04x): busy\n",
+				   mdev_uuid(mdev), schid.cssid,
+				   schid.ssid, schid.sch_no);
+		errstr = "busy";
+		goto err_out;
+	}
+
 	private->state = VFIO_CCW_STATE_CP_PROCESSING;
 	memcpy(scsw, io_region->scsw_area, sizeof(*scsw));
 

Quoted 11 lines · On Tue, 16 Jun 2020 21:50:51 +0200 Eric Farman <[email protected]> wrote:

The interrupt path checks the FSM state when processing a final interrupt (an interrupt that is neither subchannel active, nor device active), to determine whether to call cp_free() and release the associated memory. But, this does not fully close the window where a START comes in after a HALT/CLEAR. If the START runs while the CLEAR interrupt is being processed, the channel program struct will be allocated while the interrupt would be considering whether or not to free it. If the FSM state is CP_PROCESSING, then everything is fine. But if the START is able to issue its SSCH and get a cc0, then the in-flight interrupt would have been for an unrelated operation (perhaps none, if the subchannel was previously idle).

Quoted 6 lines

The channel_program struct has an "initialized" flag that is set early in the fsm_io_request() flow, to simplify the various cp_*() accessors. Let's extend this idea to include a "started" flag that announces that the channel program has successfully been issued to hardware. With this, the interrupt path can determine whether the final interrupt should also release the cp resources instead of relying on a transient FSM state.

AFAICT cp->started is potentially accessed by multiple threads, form which at least one writes. Am I right?

Actually AFAICT you want to use cp->sarted for synchronization between multiple treads (I/O requester(s), IRQ handler(s)). How does the synchronization work for bool started itself, i.e. don't we have a data race on 'started'?

A side note: I know, I asked a similar question about 'initialized' back then.

Regards,
Halil

Quoted 83 lines

Signed-off-by Eric Farman <[email protected]>

---
 drivers/s390/cio/vfio_ccw_cp.c  |  2 ++
 drivers/s390/cio/vfio_ccw_cp.h  |  1 +
 drivers/s390/cio/vfio_ccw_drv.c |  2 +-
 drivers/s390/cio/vfio_ccw_fsm.c | 11 +++++++++++
 4 files changed, 15 insertions(+), 1 deletion(-)

drivers/s390/cio/vfio_ccw_cp.c +2 -0

index b9febc581b1f..7748eeef434e 100644
@@ -657,6 +657,7 @@ int cp_init(struct channel_program *cp, struct device *mdev, union orb *orb)
 
 	if (!ret) {
 		cp->initialized = true;
+		cp->started = false;
 
 		/* It is safe to force: if it was not set but idals used
 		 * ccwchain_calc_length would have returned an error.

@@ -685,6 +686,7 @@ void cp_free(struct channel_program *cp)
 		return;
 
 	cp->initialized = false;
+	cp->started = false;
 	list_for_each_entry_safe(chain, temp, &cp->ccwchain_list, next) {
 		for (i = 0; i < chain->ch_len; i++) {
 			pfn_array_unpin_free(chain->ch_pa + i, cp->mdev);

drivers/s390/cio/vfio_ccw_cp.h +1 -0

index ba31240ce965..7ea14910aaaa 100644
@@ -39,6 +39,7 @@ struct channel_program {
 	union orb orb;
 	struct device *mdev;
 	bool initialized;
+	bool started;
 	struct ccw1 *guest_cp;
 };
 

drivers/s390/cio/vfio_ccw_drv.c +1 -1

index 8c625b530035..7e2a790dc9a1 100644
@@ -94,7 +94,7 @@ static void vfio_ccw_sch_io_todo(struct work_struct *work)
 		     (SCSW_ACTL_DEVACT | SCSW_ACTL_SCHACT));
 	if (scsw_is_solicited(&irb->scsw)) {
 		cp_update_scsw(&private->cp, &irb->scsw);
-		if (is_final && private->state == VFIO_CCW_STATE_CP_PENDING)
+		if (is_final && private->cp.started)
 			cp_free(&private->cp);
 	}
 	mutex_lock(&private->io_mutex);

drivers/s390/cio/vfio_ccw_fsm.c +11 -0

index 23e61aa638e4..d806f88eba72 100644
@@ -50,6 +50,7 @@ static int fsm_io_helper(struct vfio_ccw_private *private)
 		sch->schib.scsw.cmd.actl |= SCSW_ACTL_START_PEND;
 		ret = 0;
 		private->state = VFIO_CCW_STATE_CP_PENDING;
+		private->cp.started = true;
 		break;
 	case 1:		/* Status pending */
 	case 2:		/* Busy */

@@ -246,6 +247,16 @@ static void fsm_io_request(struct vfio_ccw_private *private,
 	char *errstr = "request";
 	struct subchannel_id schid = get_schid(private);
 
+	if (private->cp.started) {
+		io_region->ret_code = -EBUSY;
+		VFIO_CCW_MSG_EVENT(2,
+				   "%pUl (%x.%x.%04x): busy\n",
+				   mdev_uuid(mdev), schid.cssid,
+				   schid.ssid, schid.sch_no);
+		errstr = "busy";
+		goto err_out;
+	}
+
 	private->state = VFIO_CCW_STATE_CP_PROCESSING;
 	memcpy(scsw, io_region->scsw_area, sizeof(*scsw));
 

When an interrupt is received via the IRQ, the bulk of the work is stacked on a workqueue for later processing. Which means that concurrent START or a HALT/CLEAR operation (via the async_region) will race with this process and require some serialization.

Once we have all our locks acquired, let's just look to see if we're in a window where the process has been started from the IRQ, but not yet picked up by vfio-ccw to clean up an I/O. If there is, mark the request as BUSY so it can be redriven after we have a chance to breathe.

Signed-off-by Eric Farman <[email protected]>

---
 drivers/s390/cio/vfio_ccw_fsm.c | 5 +++++
 1 file changed, 5 insertions(+)

drivers/s390/cio/vfio_ccw_fsm.c +5 -0

index f0952192480e..9dc5b4d549b3 100644
@@ -28,6 +28,11 @@ static int fsm_io_helper(struct vfio_ccw_private *private)
 
 	spin_lock_irqsave(sch->lock, flags);
 
+	if (work_pending(&private->io_work)) {
+		ret = -EBUSY;
+		goto out;
+	}
+
 	orb = cp_get_orb(&private->cp, (u32)(addr_t)sch, sch->lpm);
 	if (!orb) {
 		ret = -EIO;

Let's continue our discussion of the handling of vfio-ccw interrupts.

The initial fix [1] relied upon the interrupt path's examination of the FSM state, and freeing all resources if it were CP_PENDING. But the interface used by HALT/CLEAR SUBCHANNEL doesn't affect the FSM state. Consider this sequence:

CPU 1 CPU 2
CLEAR (state=IDLE/no change)
START [2]
INTERRUPT (set state=IDLE)
INTERRUPT (set state=IDLE)

This translates to a couple of possible scenarios:

A) The START gets a cc2 because of the outstanding CLEAR, -EBUSY is returned, resources are freed, and state remains IDLE
B) The START gets a cc0 because the CLEAR has already presented an interrupt, and state is set to CP_PENDING

If the START gets a cc0 before the CLEAR INTERRUPT (stacked onto a workqueue by the IRQ context) gets a chance to run, then the INTERRUPT will release the channel program memory prematurely. If the two operations run concurrently, then the FSM state set to CP_PROCESSING will prevent the cp_free() from being invoked. But the io_mutex boundary on that path will pause itself until the START completes, and then allow the FSM to be reset to IDLE without considering the outstanding START. Neither scenario would be considered good.

Having said all of that, in v2 Conny suggested [3] the following:

Quoted 10 lines

- Detach the cp from the subchannel (or better, remove the 1:1 relationship). By that I mean building the cp as a separately allocated structure (maybe embedding a kref, but that might not be needed), and appending it to a list after SSCH with cc=0. Discard it if cc!=0.
- Remove the CP_PENDING state. The state is either IDLE after any successful SSCH/HSCH/CSCH, or a new state in that case. But no special state for SSCH.
- A successful CSCH removes the first queued request, if any.
- A final interrupt removes the first queued request, if any.

What I have implemented here is basically this, with a few changes:

- I don't queue cp's. Since there should only be one START in process at a time, and HALT/CLEAR doesn't build a cp, I didn't see a pressing need to introduce that complexity.
- Furthermore, while I initially made a separately allocated cp, adding an alloc for a cp on each I/O AND moving the guest_cp alloc from the probe path to the I/O path seems excessive. So I implemented a "started" flag to the cp, set after a cc0 from the START, and examine that on the interrupt path to determine whether cp_free() is needed.

- I opted against a "SOMETHING_PENDING" state if START/HALT/CLEAR got a cc0, and just put the FSM back to IDLE. It becomes too unwieldy to discern which operation an interrupt is completing, and whether more interrupts are expected, to be worth the additional state.
- A successful CSCH doesn't do anything special, and cp_free() is only performed on the interrupt path. Part of me wrestled with how a HALT fits into that, but mostly it was that a cc0 on any of the instructions indicated the "channel subsystem is signaled to asynchronously perform the [START/HALT/CLEAR] function." This means that an in-flight START could still receive data from the device/subchannel, so not a good idea to release memory at that point.

Separate from all that, I added a small check of the io_work queue to the FSM START path. Part of the problems I've seen was that an interrupt is presented by a CPU, but not yet processed by vfio-ccw. Some of the problems seen thus far is because of this gap, and the above changes don't address that either. Whether this is appropriate or ridiculous would be a welcome discussion.

Previous versions:
v2: https://lore.kernel.org/kvm/[email protected]/
v1: https://lore.kernel.org/kvm/[email protected]/

Footnotes:
[1] https://lore.kernel.org/kvm/62e87bf67b38dc8d5760586e7c96d400db854ebe.1562854091.git.alifm@linux.ibm.com/
[2] Halil has pointed out that QEMU should prohibit this, based on the
rules set forth by the POPs. This is true, but we should not rely on
it behaving properly without addressing this scenario that is visible
today. Once I get this behaving correctly, I'll spend some time
seeing if QEMU is misbehaving somehow.
[3] https://lore.kernel.org/kvm/[email protected]/
[4] https://lore.kernel.org/kvm/[email protected]/

Eric Farman (3):
vfio-ccw: Indicate if a channel_program is started
vfio-ccw: Remove the CP_PENDING FSM state
vfio-ccw: Check workqueue before doing START

 drivers/s390/cio/vfio_ccw_cp.c      |  2 ++
 drivers/s390/cio/vfio_ccw_cp.h      |  1 +
 drivers/s390/cio/vfio_ccw_drv.c     |  5 +----
 drivers/s390/cio/vfio_ccw_fsm.c     | 32 +++++++++++++++++------------
 drivers/s390/cio/vfio_ccw_ops.c     |  3 +--
 drivers/s390/cio/vfio_ccw_private.h |  1 -
 6 files changed, 24 insertions(+), 20 deletions(-)

Quoted 44 lines · Eric Farman

Let's continue our discussion of the handling of vfio-ccw interrupts.

The initial fix [1] relied upon the interrupt path's examination of the FSM state, and freeing all resources if it were CP_PENDING. But the interface used by HALT/CLEAR SUBCHANNEL doesn't affect the FSM state. Consider this sequence:

CPU 1 CPU 2
CLEAR (state=IDLE/no change)
START [2]
INTERRUPT (set state=IDLE)
INTERRUPT (set state=IDLE)

This translates to a couple of possible scenarios:

A) The START gets a cc2 because of the outstanding CLEAR, -EBUSY is returned, resources are freed, and state remains IDLE
B) The START gets a cc0 because the CLEAR has already presented an interrupt, and state is set to CP_PENDING

If the START gets a cc0 before the CLEAR INTERRUPT (stacked onto a workqueue by the IRQ context) gets a chance to run, then the INTERRUPT will release the channel program memory prematurely. If the two operations run concurrently, then the FSM state set to CP_PROCESSING will prevent the cp_free() from being invoked. But the io_mutex boundary on that path will pause itself until the START completes, and then allow the FSM to be reset to IDLE without considering the outstanding START. Neither scenario would be considered good.

Having said all of that, in v2 Conny suggested [3] the following:

Quoted 10 lines

- Detach the cp from the subchannel (or better, remove the 1:1 relationship). By that I mean building the cp as a separately allocated structure (maybe embedding a kref, but that might not be needed), and appending it to a list after SSCH with cc=0. Discard it if cc!=0.
- Remove the CP_PENDING state. The state is either IDLE after any successful SSCH/HSCH/CSCH, or a new state in that case. But no special state for SSCH.
- A successful CSCH removes the first queued request, if any.
- A final interrupt removes the first queued request, if any.

What I have implemented here is basically this, with a few changes:

Quoted 8 lines

- I don't queue cp's. Since there should only be one START in process at a time, and HALT/CLEAR doesn't build a cp, I didn't see a pressing need to introduce that complexity.
- Furthermore, while I initially made a separately allocated cp, adding an alloc for a cp on each I/O AND moving the guest_cp alloc from the probe path to the I/O path seems excessive. So I implemented a "started" flag to the cp, set after a cc0 from the START, and examine that on the interrupt path to determine whether cp_free() is needed.

FYI... After a day or two of running, I sprung a kernel debug oops for list corruption in ccwchain_free(). I'm going to blame this piece, since it was the last thing I changed and I hadn't come across any such damage since v2. So either "started" is a bad idea, or a broken one. Or both. :)

Quoted 54 lines · On Wed, 17 Jun 2020 07:24:17 -0400 Eric Farman <[email protected]> wrote:

Quoted 52 lines · Eric Farman

Let's continue our discussion of the handling of vfio-ccw interrupts.

The initial fix [1] relied upon the interrupt path's examination of the FSM state, and freeing all resources if it were CP_PENDING. But the interface used by HALT/CLEAR SUBCHANNEL doesn't affect the FSM state. Consider this sequence:

CPU 1 CPU 2
CLEAR (state=IDLE/no change)
START [2]
INTERRUPT (set state=IDLE)
INTERRUPT (set state=IDLE)

This translates to a couple of possible scenarios:

A) The START gets a cc2 because of the outstanding CLEAR, -EBUSY is returned, resources are freed, and state remains IDLE
B) The START gets a cc0 because the CLEAR has already presented an interrupt, and state is set to CP_PENDING

If the START gets a cc0 before the CLEAR INTERRUPT (stacked onto a workqueue by the IRQ context) gets a chance to run, then the INTERRUPT will release the channel program memory prematurely. If the two operations run concurrently, then the FSM state set to CP_PROCESSING will prevent the cp_free() from being invoked. But the io_mutex boundary on that path will pause itself until the START completes, and then allow the FSM to be reset to IDLE without considering the outstanding START. Neither scenario would be considered good.

Having said all of that, in v2 Conny suggested [3] the following:

Quoted 10 lines

- Detach the cp from the subchannel (or better, remove the 1:1 relationship). By that I mean building the cp as a separately allocated structure (maybe embedding a kref, but that might not be needed), and appending it to a list after SSCH with cc=0. Discard it if cc!=0.
- Remove the CP_PENDING state. The state is either IDLE after any successful SSCH/HSCH/CSCH, or a new state in that case. But no special state for SSCH.
- A successful CSCH removes the first queued request, if any.
- A final interrupt removes the first queued request, if any.

What I have implemented here is basically this, with a few changes:

- I don't queue cp's. Since there should only be one START in process at a time, and HALT/CLEAR doesn't build a cp, I didn't see a pressing need to introduce that complexity.
- Furthermore, while I initially made a separately allocated cp, adding an alloc for a cp on each I/O AND moving the guest_cp alloc from the probe path to the I/O path seems excessive. So I implemented a "started" flag to the cp, set after a cc0 from the START, and examine that on the interrupt path to determine whether cp_free() is needed.

Quoted 4 lines

FYI... After a day or two of running, I sprung a kernel debug oops for list corruption in ccwchain_free(). I'm going to blame this piece, since it was the last thing I changed and I hadn't come across any such damage since v2. So either "started" is a bad idea, or a broken one. Or both. :)

Have you come to any conclusion wrt 'started'? Not wanting to generate stress, just asking :)

Let's continue our discussion of the handling of vfio-ccw interrupts.

The initial fix [1] relied upon the interrupt path's examination of the FSM state, and freeing all resources if it were CP_PENDING. But the interface used by HALT/CLEAR SUBCHANNEL doesn't affect the FSM state. Consider this sequence:

CPU 1 CPU 2
CLEAR (state=IDLE/no change)
START [2]
INTERRUPT (set state=IDLE)
INTERRUPT (set state=IDLE)

This translates to a couple of possible scenarios:

A) The START gets a cc2 because of the outstanding CLEAR, -EBUSY is returned, resources are freed, and state remains IDLE
B) The START gets a cc0 because the CLEAR has already presented an interrupt, and state is set to CP_PENDING

If the START gets a cc0 before the CLEAR INTERRUPT (stacked onto a workqueue by the IRQ context) gets a chance to run, then the INTERRUPT will release the channel program memory prematurely. If the two operations run concurrently, then the FSM state set to CP_PROCESSING will prevent the cp_free() from being invoked. But the io_mutex boundary on that path will pause itself until the START completes, and then allow the FSM to be reset to IDLE without considering the outstanding START. Neither scenario would be considered good.

Having said all of that, in v2 Conny suggested [3] the following:

Quoted 10 lines

- Detach the cp from the subchannel (or better, remove the 1:1 relationship). By that I mean building the cp as a separately allocated structure (maybe embedding a kref, but that might not be needed), and appending it to a list after SSCH with cc=0. Discard it if cc!=0.
- Remove the CP_PENDING state. The state is either IDLE after any successful SSCH/HSCH/CSCH, or a new state in that case. But no special state for SSCH.
- A successful CSCH removes the first queued request, if any.
- A final interrupt removes the first queued request, if any.

What I have implemented here is basically this, with a few changes:

- I don't queue cp's. Since there should only be one START in process at a time, and HALT/CLEAR doesn't build a cp, I didn't see a pressing need to introduce that complexity.
- Furthermore, while I initially made a separately allocated cp, adding an alloc for a cp on each I/O AND moving the guest_cp alloc from the probe path to the I/O path seems excessive. So I implemented a "started" flag to the cp, set after a cc0 from the START, and examine that on the interrupt path to determine whether cp_free() is needed.
- I opted against a "SOMETHING_PENDING" state if START/HALT/CLEAR got a cc0, and just put the FSM back to IDLE. It becomes too unwieldy to discern which operation an interrupt is completing, and whether

more interrupts are expected, to be worth the additional state.
- A successful CSCH doesn't do anything special, and cp_free() is only performed on the interrupt path. Part of me wrestled with how a HALT fits into that, but mostly it was that a cc0 on any of the instructions indicated the "channel subsystem is signaled to asynchronously perform the [START/HALT/CLEAR] function." This means that an in-flight START could still receive data from the device/subchannel, so not a good idea to release memory at that point.

Separate from all that, I added a small check of the io_work queue to the FSM START path. Part of the problems I've seen was that an interrupt is presented by a CPU, but not yet processed by vfio-ccw. Some of the problems seen thus far is because of this gap, and the above changes don't address that either. Whether this is appropriate or ridiculous would be a welcome discussion.

Previous versions:
v2: https://lore.kernel.org/kvm/[email protected]/
v1: https://lore.kernel.org/kvm/[email protected]/

Footnotes:
[1] https://lore.kernel.org/kvm/62e87bf67b38dc8d5760586e7c96d400db854ebe.1562854091.git.alifm@linux.ibm.com/
[2] Halil has pointed out that QEMU should prohibit this, based on the
rules set forth by the POPs. This is true, but we should not rely on
it behaving properly without addressing this scenario that is visible
today. Once I get this behaving correctly, I'll spend some time
seeing if QEMU is misbehaving somehow.
[3] https://lore.kernel.org/kvm/[email protected]/
[4] https://lore.kernel.org/kvm/[email protected]/

Eric Farman (3):
vfio-ccw: Indicate if a channel_program is started
vfio-ccw: Remove the CP_PENDING FSM state
vfio-ccw: Check workqueue before doing START

 drivers/s390/cio/vfio_ccw_cp.c      |  2 ++
 drivers/s390/cio/vfio_ccw_cp.h      |  1 +
 drivers/s390/cio/vfio_ccw_drv.c     |  5 +----
 drivers/s390/cio/vfio_ccw_fsm.c     | 32 +++++++++++++++++------------
 drivers/s390/cio/vfio_ccw_ops.c     |  3 +--
 drivers/s390/cio/vfio_ccw_private.h |  1 -
 6 files changed, 24 insertions(+), 20 deletions(-)