Skip to contentSkip to conversation

s390x/vfio: VFIO-AP interrupt control interception

RFC PATCH New Pierre Morel 3 participants 6 messages Feb 14, 2019 – Feb 15, 2019

lore
Share & tools
lore
#0 lore raw

RFC PATCH v3 1/1 s390x/vfio: VFIO-AP interrupt control interception

This QEMU patch is to be used in conjonction with the following patch series in LINUX.

vfio: ap: ioctl definitions for AP Queue Interrupt Control

The LINUX patch series handle all the interception in KVM and we do not need to take care of the interception in QEMU.

The goal of this patch is to define the apqi CPU feature which will be proposed by KVM when the AP Interrupt Control facility is to be available in the guest.

This feature must be turned on by QEMU to intercept AP instructions on the KVM guest.
The QEMU command line to turn this feature on looks something like:

    qemu-system-s390x ... -cpu xxx,apqi=on ...

Pierre Morel (1):
s390x/cpumodel: Set up CPU model for AQIC interception

 target/s390x/cpu_features.c     | 1 +
 target/s390x/cpu_features_def.h | 1 +
 target/s390x/cpu_models.c       | 1 +
 target/s390x/gen-features.c     | 1 +
 4 files changed, 4 insertions(+)
Signature · 1 line
2.7.4
#1 lore raw

PATCH v3 1/1 s390x/cpumodel: Set up CPU model for AQIC interception

A new CPU model facilities is introduced to support AP devices interruption interception for a KVM guest.

"APQI" for "AP-Queue Interruption" facility

The S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL, CPU facility indicates whether the PQAP instruction with the AQIC command is available to the guest.
This feature will be enabled only if the AP instructions are available on the linux host and AQIC facility is installed on the host.

This feature must be turned on from userspace to intercept AP instructions on the KVM guest. The QEMU command line to turn this feature on looks something like this:

    qemu-system-s390x ... -cpu xxx,apqi=on ...
---
 target/s390x/cpu_features.c     | 1 +
 target/s390x/cpu_features_def.h | 1 +
 target/s390x/cpu_models.c       | 1 +
 target/s390x/gen-features.c     | 1 +
 4 files changed, 4 insertions(+)
target/s390x/cpu_features.c +1 -0
index 60cfeba..99bd382 100644
@@ -84,6 +84,7 @@ static const S390FeatDef s390_features[] = {
     FEAT_INIT("sema", S390_FEAT_TYPE_STFL, 59, "Semaphore-assist facility"),
     FEAT_INIT("tsi", S390_FEAT_TYPE_STFL, 60, "Time-slice Instrumentation facility"),
     FEAT_INIT("ri", S390_FEAT_TYPE_STFL, 64, "CPU runtime-instrumentation facility"),
+    FEAT_INIT("apqi", S390_FEAT_TYPE_STFL, 65, "AP-Queue interruption facility"),
     FEAT_INIT("zpci", S390_FEAT_TYPE_STFL, 69, "z/PCI facility"),
     FEAT_INIT("aen", S390_FEAT_TYPE_STFL, 71, "General-purpose-adapter-event-notification facility"),
     FEAT_INIT("ais", S390_FEAT_TYPE_STFL, 72, "General-purpose-adapter-interruption-suppression facility"),
target/s390x/cpu_features_def.h +1 -0
index 5fc7e7b..3f22780 100644
@@ -72,6 +72,7 @@ typedef enum {
     S390_FEAT_SEMAPHORE_ASSIST,
     S390_FEAT_TIME_SLICE_INSTRUMENTATION,
     S390_FEAT_RUNTIME_INSTRUMENTATION,
+    S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL,
     S390_FEAT_ZPCI,
     S390_FEAT_ADAPTER_EVENT_NOTIFICATION,
     S390_FEAT_ADAPTER_INT_SUPPRESSION,
target/s390x/cpu_models.c +1 -0
index 7c253ff..6b5e94b 100644
@@ -788,6 +788,7 @@ static void check_consistency(const S390CPUModel *model)
         { S390_FEAT_SIE_KSS, S390_FEAT_SIE_F2 },
         { S390_FEAT_AP_QUERY_CONFIG_INFO, S390_FEAT_AP },
         { S390_FEAT_AP_FACILITIES_TEST, S390_FEAT_AP },
+        { S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL, S390_FEAT_AP },
     };
     int i;
 
target/s390x/gen-features.c +1 -0
index 70015ea..b051221 100644
@@ -448,6 +448,7 @@ static uint16_t full_GEN12_GA1[] = {
     S390_FEAT_EDAT_2,
     S390_FEAT_SIDE_EFFECT_ACCESS_ESOP2,
     S390_FEAT_AP_QUERY_CONFIG_INFO,
+    S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL,
     S390_FEAT_AP_FACILITIES_TEST,
     S390_FEAT_AP,
 };
Signature · 1 line
2.7.4
#2 lore raw

Re: [Qemu-devel] [PATCH v3] s390x/cpumodel: Set up CPU model for AQIC interception

Quoted 12 lines · Pierre Morel

A new CPU model facilities is introduced to support AP devices interruption interception for a KVM guest.

"APQI" for "AP-Queue Interruption" facility

The S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL, CPU facility indicates whether the PQAP instruction with the AQIC command is available to the guest.
This feature will be enabled only if the AP instructions are available on the linux host and AQIC facility is installed on the host.

Quoted 5 lines

This feature must be turned on from userspace to intercept AP instructions on the KVM guest. The QEMU command line to turn this feature on looks something like this:

     qemu-system-s390x ... -cpu xxx,apqi=on ...

Hi,

I suddenly have a very big doubt on this QEMU patch.

Older Linux advertise PQAP/AQIC feature (65)
Older Qemu mask it for the guest

Now if we set aqic = on in the new QEMU,
QEMU advertises feature (65)

The guest gets an OPERATION EXCEPTION even we advertised the feature 65.

This seems wrong to me, we should verify that the vfio_driver is loaded in the host before starting the guest with something like:

kvm_vm_check_attr(kvm_state, KVM_S390_VM_ENABLE_AQIC)

Don't you think?

Regards,

Pierre

Quoted 61 lines
---
  target/s390x/cpu_features.c     | 1 +
  target/s390x/cpu_features_def.h | 1 +
  target/s390x/cpu_models.c       | 1 +
  target/s390x/gen-features.c     | 1 +
  4 files changed, 4 insertions(+)
target/s390x/cpu_features.c +1 -0
index 60cfeba..99bd382 100644
@@ -84,6 +84,7 @@ static const S390FeatDef s390_features[] = {
      FEAT_INIT("sema", S390_FEAT_TYPE_STFL, 59, "Semaphore-assist facility"),
      FEAT_INIT("tsi", S390_FEAT_TYPE_STFL, 60, "Time-slice Instrumentation facility"),
      FEAT_INIT("ri", S390_FEAT_TYPE_STFL, 64, "CPU runtime-instrumentation facility"),
+    FEAT_INIT("apqi", S390_FEAT_TYPE_STFL, 65, "AP-Queue interruption facility"),
      FEAT_INIT("zpci", S390_FEAT_TYPE_STFL, 69, "z/PCI facility"),
      FEAT_INIT("aen", S390_FEAT_TYPE_STFL, 71, "General-purpose-adapter-event-notification facility"),
      FEAT_INIT("ais", S390_FEAT_TYPE_STFL, 72, "General-purpose-adapter-interruption-suppression facility"),
target/s390x/cpu_features_def.h +1 -0
index 5fc7e7b..3f22780 100644
@@ -72,6 +72,7 @@ typedef enum {
      S390_FEAT_SEMAPHORE_ASSIST,
      S390_FEAT_TIME_SLICE_INSTRUMENTATION,
      S390_FEAT_RUNTIME_INSTRUMENTATION,
+    S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL,
      S390_FEAT_ZPCI,
      S390_FEAT_ADAPTER_EVENT_NOTIFICATION,
      S390_FEAT_ADAPTER_INT_SUPPRESSION,
target/s390x/cpu_models.c +1 -0
index 7c253ff..6b5e94b 100644
@@ -788,6 +788,7 @@ static void check_consistency(const S390CPUModel *model)
          { S390_FEAT_SIE_KSS, S390_FEAT_SIE_F2 },
          { S390_FEAT_AP_QUERY_CONFIG_INFO, S390_FEAT_AP },
          { S390_FEAT_AP_FACILITIES_TEST, S390_FEAT_AP },
+        { S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL, S390_FEAT_AP },
      };
      int i;
 
target/s390x/gen-features.c +1 -0
index 70015ea..b051221 100644
@@ -448,6 +448,7 @@ static uint16_t full_GEN12_GA1[] = {
      S390_FEAT_EDAT_2,
      S390_FEAT_SIDE_EFFECT_ACCESS_ESOP2,
      S390_FEAT_AP_QUERY_CONFIG_INFO,
+    S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL,
      S390_FEAT_AP_FACILITIES_TEST,
      S390_FEAT_AP,
  };
Signature · 2 lines
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany
#3 lore raw

Re: [Qemu-devel] [PATCH v3] s390x/cpumodel: Set up CPU model for AQIC interception

Quoted 21 lines · Pierre Morel
Quoted 12 lines · Pierre Morel

A new CPU model facilities is introduced to support AP devices interruption interception for a KVM guest.

"APQI" for "AP-Queue Interruption" facility

The S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL, CPU facility indicates whether the PQAP instruction with the AQIC command is available to the guest.
This feature will be enabled only if the AP instructions are available on the linux host and AQIC facility is installed on the host.

Quoted 5 lines

This feature must be turned on from userspace to intercept AP instructions on the KVM guest. The QEMU command line to turn this feature on looks something like this:

     qemu-system-s390x ... -cpu xxx,apqi=on ...

Quoted 5 lines

Hi,

I suddenly have a very big doubt on this QEMU patch.

Older Linux advertise PQAP/AQIC feature (65)

Which Linuxes advertise PQAP/AQIC (stfle.65)?
This was never advertised by any previous Linux as far as I can tell.

#4 lore raw

Re: [Qemu-devel] [PATCH v3] s390x/cpumodel: Set up CPU model for AQIC interception

Quoted 24 lines · Christian Borntraeger
Quoted 21 lines · Pierre Morel
Quoted 17 lines · Pierre Morel

A new CPU model facilities is introduced to support AP devices interruption interception for a KVM guest.

"APQI" for "AP-Queue Interruption" facility

The S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL, CPU facility indicates whether the PQAP instruction with the AQIC command is available to the guest.
This feature will be enabled only if the AP instructions are available on the linux host and AQIC facility is installed on the host.

This feature must be turned on from userspace to intercept AP instructions on the KVM guest. The QEMU command line to turn this feature on looks something like this:

     qemu-system-s390x ... -cpu xxx,apqi=on ...

Quoted 9 lines
Quoted 5 lines

Hi,

I suddenly have a very big doubt on this QEMU patch.

Older Linux advertise PQAP/AQIC feature (65)

Which Linuxes advertise PQAP/AQIC (stfle.65)?
This was never advertised by any previous Linux as far as I can tell.

OK, very sorry, you are right, I checked on a different machine and it is not. I must have left something in my system.

Sorry for the panic.
I will look at what made me experience this.

Regards,
Pierre

Signature · 2 lines
Pierre Morel
Linux/KVM/QEMU in Böblingen - Germany
#5 lore raw

Re: [Qemu-devel] [PATCH v3] s390x/cpumodel: Set up CPU model for AQIC interception

In reply to Pierre Morel
Quoted 2 lines · Pierre Morel

A new CPU model facilities is introduced to support AP devices interruption interception for a KVM guest.

This sentence is confusing. At first I thought you would introduce a new _fake_ facility.

"Let's add support for the AP-Queue interruption facility to the CPU model".

Quoted 69 lines

"APQI" for "AP-Queue Interruption" facility

The S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL, CPU facility indicates whether the PQAP instruction with the AQIC command is available to the guest.
This feature will be enabled only if the AP instructions are available on the linux host and AQIC facility is installed on the host.

This feature must be turned on from userspace to intercept AP instructions on the KVM guest. The QEMU command line to turn this feature on looks something like this:

    qemu-system-s390x ... -cpu xxx,apqi=on ...
---
 target/s390x/cpu_features.c     | 1 +
 target/s390x/cpu_features_def.h | 1 +
 target/s390x/cpu_models.c       | 1 +
 target/s390x/gen-features.c     | 1 +
 4 files changed, 4 insertions(+)
target/s390x/cpu_features.c +1 -0
index 60cfeba..99bd382 100644
@@ -84,6 +84,7 @@ static const S390FeatDef s390_features[] = {
     FEAT_INIT("sema", S390_FEAT_TYPE_STFL, 59, "Semaphore-assist facility"),
     FEAT_INIT("tsi", S390_FEAT_TYPE_STFL, 60, "Time-slice Instrumentation facility"),
     FEAT_INIT("ri", S390_FEAT_TYPE_STFL, 64, "CPU runtime-instrumentation facility"),
+    FEAT_INIT("apqi", S390_FEAT_TYPE_STFL, 65, "AP-Queue interruption facility"),
     FEAT_INIT("zpci", S390_FEAT_TYPE_STFL, 69, "z/PCI facility"),
     FEAT_INIT("aen", S390_FEAT_TYPE_STFL, 71, "General-purpose-adapter-event-notification facility"),
     FEAT_INIT("ais", S390_FEAT_TYPE_STFL, 72, "General-purpose-adapter-interruption-suppression facility"),
target/s390x/cpu_features_def.h +1 -0
index 5fc7e7b..3f22780 100644
@@ -72,6 +72,7 @@ typedef enum {
     S390_FEAT_SEMAPHORE_ASSIST,
     S390_FEAT_TIME_SLICE_INSTRUMENTATION,
     S390_FEAT_RUNTIME_INSTRUMENTATION,
+    S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL,
     S390_FEAT_ZPCI,
     S390_FEAT_ADAPTER_EVENT_NOTIFICATION,
     S390_FEAT_ADAPTER_INT_SUPPRESSION,
target/s390x/cpu_models.c +1 -0
index 7c253ff..6b5e94b 100644
@@ -788,6 +788,7 @@ static void check_consistency(const S390CPUModel *model)
         { S390_FEAT_SIE_KSS, S390_FEAT_SIE_F2 },
         { S390_FEAT_AP_QUERY_CONFIG_INFO, S390_FEAT_AP },
         { S390_FEAT_AP_FACILITIES_TEST, S390_FEAT_AP },
+        { S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL, S390_FEAT_AP },
     };
     int i;
 
target/s390x/gen-features.c +0 -0
index 70015ea..b051221 100644

target/s390x/gen-features.c static uint16_t full_GEN12_GA1[] = { · line 454

Quoted 7 lines
target/s390x/gen-features.c +1 -0
@@ -448,6 +448,7 @@ static uint16_t full_GEN12_GA1[] = {
     S390_FEAT_EDAT_2,
     S390_FEAT_SIDE_EFFECT_ACCESS_ESOP2,
     S390_FEAT_AP_QUERY_CONFIG_INFO,
+    S390_FEAT_AP_QUEUE_INTERRUPT_CONTROL,
     S390_FEAT_AP_FACILITIES_TEST,
     S390_FEAT_AP,
 };

Looks good to me. In QEMU, we enable/indicate facilities only when all required parts were implemented on the QEMU side.

Is there anything we have to do regarding
- migration support unlocked with this facility
- functional changes we have to implement?

Or is this really a "kernel does it all, migration not relevant" ?

Signature · 3 lines
Thanks,

David / dhildenb