vfio-ccw: forward halt/clear errors
RFC PATCH 1/1 vfio-ccw: forward halt/clear errors
hsch and csch basically have two parts: execute the command, and perform the halt/clear function. For fully emulated subchannels, it is pretty clear how it will work: check the subchannel state, and actually 'perform the halt/clear function' and set cc 0 if everything looks good.
For passthrough subchannels, some of the checking is done within QEMU, but some has to be done within the kernel. QEMU's subchannel state may be such that we can perform the async function, but the kernel may still get a cc != 0 when it is actually executing the instruction. In that case, we need to set the condition actually encountered by the kernel; if we set cc 0 on error, we would actually need to inject an interrupt as well.
---
Stumbled over this during the vfio-ccw kernel locking discussions.
This is probably a corner case, and I'm not sure how I can actually get this path excercised, but it passes my smoke tests.
Not sure whether this is the way to go. The unit exceptions in the halt/clear error paths also seem slightly fishy.
--- hw/s390x/css.c | 34 ++++++++++++++++++++++++++++++---- hw/vfio/ccw.c | 4 ++-- 2 files changed, 32 insertions(+), 6 deletions(-)
hw/s390x/css.c +30 -4
@@ -1206,23 +1206,49 @@ static void sch_handle_start_func_virtual(SubchDev *sch) } -static void sch_handle_halt_func_passthrough (SubchDev *sch) +static IOInstEnding sch_handle_halt_func_passthrough (SubchDev *sch) {int ret; ret =s390_ccw_halt (sch);if (ret == -ENOSYS ) {sch_handle_halt_func (sch); +return IOINST_CC_EXPECTED ; + } +/* +* Some conditions may have been detected prior to starting the halt +* function; map them to the correct cc. +*/ +switch (ret) { +case -EBUSY : +return IOINST_CC_BUSY ; +case -ENODEV : +case -EACCES : +return IOINST_CC_NOT_OPERATIONAL ; +default : +return IOINST_CC_EXPECTED ; } } -static void sch_handle_clear_func_passthrough (SubchDev *sch) +static IOInstEnding sch_handle_clear_func_passthrough (SubchDev *sch) {int ret; ret =s390_ccw_clear (sch);if (ret == -ENOSYS ) {sch_handle_clear_func (sch); +return IOINST_CC_EXPECTED ; + } +/* +* Some conditions may have been detected prior to starting the clear +* function; map them to the correct cc. +*/ +switch (ret) { +case -ENODEV : +case -EACCES : +return IOINST_CC_NOT_OPERATIONAL ; +default : +return IOINST_CC_EXPECTED ; } }
@@ -1265,9 +1291,9 @@ IOInstEnding do_subchannel_work_passthrough(SubchDev *sch)SCHIB *schib = &sch->curr_status;if (schib->scsw.ctrl &SCSW_FCTL_CLEAR_FUNC ) { -sch_handle_clear_func_passthrough (sch); +return sch_handle_clear_func_passthrough (sch); }else if (schib->scsw.ctrl &SCSW_FCTL_HALT_FUNC ) { -sch_handle_halt_func_passthrough (sch); +return sch_handle_halt_func_passthrough (sch); }else if (schib->scsw.ctrl &SCSW_FCTL_START_FUNC ) {return sch_handle_start_func_passthrough (sch); }
hw/vfio/ccw.c +2 -2
@@ -199,7 +199,7 @@ again:case 0 :case -ENODEV :case -EACCES : -return 0 ; +return ret;case -EFAULT :default :sch_gen_unit_exception (sch);
@@ -240,7 +240,7 @@ again:case -EBUSY :case -ENODEV :case -EACCES : -return 0 ; +return ret;case -EFAULT :default :sch_gen_unit_exception (sch);
Signature · 1 line
2.31.1
Quoted 15 lines · Cornelia Huck
hsch and csch basically have two parts: execute the command, and perform the halt/clear function. For fully emulated subchannels, it is pretty clear how it will work: check the subchannel state, and actually 'perform the halt/clear function' and set cc 0 if everything looks good.
For passthrough subchannels, some of the checking is done within QEMU, but some has to be done within the kernel. QEMU's subchannel state may be such that we can perform the async function, but the kernel may still get a cc != 0 when it is actually executing the instruction. In that case, we need to set the condition actually encountered by the kernel; if we set cc 0 on error, we would actually need to inject an interrupt as well.
Quoted 7 lines
Signed-off-by Cornelia Huck <[email protected]>---
Stumbled over this during the vfio-ccw kernel locking discussions.
This is probably a corner case, and I'm not sure how I can actually get this path excercised, but it passes my smoke tests.
I'll see if I can hammer my way into some of this.
Quoted 2 lines
Not sure whether this is the way to go.
I think it seems reasonable.
Quoted 2 lines
The unit exceptions in the
halt/clear error paths also seem slightly fishy.
It is peculiar. Looking at the old POPS references, the unit exception states that the --device-- detected something unusual, not really the subchannel (which is how vfio-ccw is behaving). But, providing some indication that something went seriously wrong is good. Which I guess was the point of the UE code, even though it's obviously set up to be generated after a failure on the START.
I guess at the least, we need to clean up the FCTL based on the function that failed, rather than only cleaning up the START function. The UE itself may just be an extra "hey this is busted" indicator.
Quoted 14 lines
--- hw/s390x/css.c | 34 ++++++++++++++++++++++++++++++---- hw/vfio/ccw.c | 4 ++-- 2 files changed, 32 insertions(+), 6 deletions(-)hw/s390x/css.c +0
-0@@ -1206,23 +1206,49 @@ static void sch_handle_start_func_virtual(SubchDev *sch) }
hw/s390x/css.c static void · line 1217
Quoted 10 lines
hw/s390x/css.c +2
-1-static void sch_handle_halt_func_passthrough (SubchDev *sch) +static IOInstEnding sch_handle_halt_func_passthrough (SubchDev *sch) {int ret; ret =s390_ccw_halt (sch);if (ret == -ENOSYS ) {sch_handle_halt_func (sch); +return IOINST_CC_EXPECTED ;
This is the fallback, so makes sense. You could fold it into the switch table, but since that's for "stuff from the kernel" versus -ENOSYS says "there's no way to call the kernel" I guess this is fine too.
hw/vfio/ccw.c again:
Quoted 59 lines
hw/s390x/css.c +3
-0+ } +/* +* Some conditions may have been detected prior to starting the halt
hw/s390x/css.c +15
-1+* function; map them to the correct cc. +*/ +switch (ret) { +case -EBUSY : +return IOINST_CC_BUSY ; +case -ENODEV : +case -EACCES : +return IOINST_CC_NOT_OPERATIONAL ; +default : +return IOINST_CC_EXPECTED ; } } -static void sch_handle_clear_func_passthrough (SubchDev *sch) +static IOInstEnding sch_handle_clear_func_passthrough (SubchDev *sch) {int ret; ret =s390_ccw_clear (sch);if (ret == -ENOSYS ) {sch_handle_clear_func (sch); +return IOINST_CC_EXPECTED ; + } +/* +* Some conditions may have been detected prior to starting the clear
hw/s390x/css.c +10
-2+* function; map them to the correct cc. +*/ +switch (ret) { +case -ENODEV : +case -EACCES : +return IOINST_CC_NOT_OPERATIONAL ; +default : +return IOINST_CC_EXPECTED ; } }@@ -1265,9 +1291,9 @@ IOInstEndingdo_subchannel_work_passthrough (SubchDev *sch) SCHIB *schib = &sch->curr_status;if (schib->scsw.ctrl &SCSW_FCTL_CLEAR_FUNC ) { -sch_handle_clear_func_passthrough (sch); +return sch_handle_clear_func_passthrough (sch); }else if (schib->scsw.ctrl &SCSW_FCTL_HALT_FUNC ) { -sch_handle_halt_func_passthrough (sch); +return sch_handle_halt_func_passthrough (sch); }else if (schib->scsw.ctrl &SCSW_FCTL_START_FUNC ) {return sch_handle_start_func_passthrough (sch);}hw/vfio/ccw.c +0
-0@@ -199,7 +199,7 @@ again:
// This is for CLEAR
hw/vfio/ccw.c again:
Quoted 9 lines
hw/vfio/ccw.c +1
-1case 0 :case -ENODEV :case -EACCES : -return 0 ; +return ret;case -EFAULT :default :sch_gen_unit_exception (sch);@@ -240,7 +240,7 @@ again:
// This is for HALT
hw/vfio/ccw.c again: · line 243
Quoted 5 lines
hw/vfio/ccw.c +1
-1case -EBUSY :case -ENODEV :case -EACCES : -return 0 ; +return ret;
Aside: How could we get EACCES for either HALT or CLEAR? I only see that set in the normal request path, if we got a CC3 on the SSCH.
Can we scrub them, or do we need to update kernelDocumentation/s390/vfio-ccw.rst ? :)
Eric
Quoted 3 lines
hw/vfio/ccw.c +0
-0case -EFAULT :default :sch_gen_unit_exception (sch);
Quoted 19 lines · On Wed, 12 May 2021 16:19:15 -0400 Eric Farman <[email protected]> wrote:
Quoted 18 lines · Cornelia Huck
hsch and csch basically have two parts: execute the command, and perform the halt/clear function. For fully emulated subchannels, it is pretty clear how it will work: check the subchannel state, and actually 'perform the halt/clear function' and set cc 0 if everything looks good.
For passthrough subchannels, some of the checking is done within QEMU, but some has to be done within the kernel. QEMU's subchannel state may be such that we can perform the async function, but the kernel may still get a cc != 0 when it is actually executing the instruction. In that case, we need to set the condition actually encountered by the kernel; if we set cc 0 on error, we would actually need to inject an interrupt as well.
Signed-off-by Cornelia Huck <[email protected]>---
Quoted 6 lines
Quoted 4 lines
Stumbled over this during the vfio-ccw kernel locking discussions.
This is probably a corner case, and I'm not sure how I can actually get this path excercised, but it passes my smoke tests.
I'll see if I can hammer my way into some of this.
Thanks, that would be cool.
Quoted 9 lines
Quoted 3 lines
Not sure whether this is the way to go.
I think it seems reasonable.
Quoted 2 lines
The unit exceptions in the
halt/clear error paths also seem slightly fishy.
Quoted 6 lines
It is peculiar. Looking at the old POPS references, the unit exception states that the --device-- detected something unusual, not really the subchannel (which is how vfio-ccw is behaving). But, providing some indication that something went seriously wrong is good. Which I guess was the point of the UE code, even though it's obviously set up to be generated after a failure on the START.
Yes, I had copied it over when I wired up halt/clear.
Quoted 3 lines
I guess at the least, we need to clean up the FCTL based on the function that failed, rather than only cleaning up the START function.
Makes sense.
Quoted 1 line
The UE itself may just be an extra "hey this is busted" indicator.
It still feels a bit odd, but I'm not sure that there's a better alternative.
Quoted 21 lines
Quoted 21 lines
--- hw/s390x/css.c | 34 ++++++++++++++++++++++++++++++---- hw/vfio/ccw.c | 4 ++-- 2 files changed, 32 insertions(+), 6 deletions(-)hw/s390x/css.c +1
-1@@ -1206,23 +1206,49 @@ static void sch_handle_start_func_virtual(SubchDev *sch) } -static void sch_handle_halt_func_passthrough (SubchDev *sch) +static IOInstEnding sch_handle_halt_func_passthrough (SubchDev *sch) {int ret;
Quoted 8 lines
hw/s390x/css.c static void sch_handle_start_func_virtual(SubchDev *sch) · line 1216
Quoted 4 lines
hw/s390x/css.c +1
-0ret =s390_ccw_halt (sch);if (ret == -ENOSYS ) {sch_handle_halt_func (sch); +return IOINST_CC_EXPECTED ;This is the fallback, so makes sense. You could fold it into the switch table, but since that's for "stuff from the kernel" versus
-ENOSYSsays "there's no way to call the kernel" I guess this is fine too.
Yes, that was the reason I kept that separate. (I don't expect it to be a heavily exercised path nowadays anyway.)
Quoted 17 lines
Quoted 17 lines
hw/s390x/css.c +3
-0+ } +/* +* Some conditions may have been detected prior to starting the halt
hw/s390x/css.c +10
-0+* function; map them to the correct cc. +*/ +switch (ret) { +case -EBUSY : +return IOINST_CC_BUSY ; +case -ENODEV : +case -EACCES : +return IOINST_CC_NOT_OPERATIONAL ; +default : +return IOINST_CC_EXPECTED ; } }
(...)
Quoted 20 lines
hw/vfio/ccw.c again:
Quoted 5 lines
hw/vfio/ccw.c +0
-0@@ -199,7 +199,7 @@ again:// This is for CLEAR
hw/vfio/ccw.c again:
Quoted 9 lines
hw/vfio/ccw.c +1
-1case 0 :case -ENODEV :case -EACCES : -return 0 ; +return ret;case -EFAULT :default :sch_gen_unit_exception (sch);@@ -240,7 +240,7 @@ again:// This is for HALT
Quoted 8 lines
hw/vfio/ccw.c again: · line 243
Quoted 5 lines
hw/vfio/ccw.c +1
-1case -EBUSY :case -ENODEV :case -EACCES : -return 0 ; +return ret;Aside: How could we get EACCES for either HALT or CLEAR? I only see that set in the normal request path, if we got a CC3 on the SSCH.
We should only get -ENODEV, which basically indicates cc3 on the kernel side. For start, the kernel distinguishes between "you didn't specify a path in the orb that's actually available" and "device not operational", but we map everything to cc 3 in QEMU (and I don't think there's anything else we could do with that information anyway.)
Quoted 3 lines
Can we scrub them, or do we need to update kernel
Documentation/s390/vfio-ccw.rst? :)
We could remove them, or log something if they come up unexpectedly; but their presence does not really hurt, either.