Skip to contentSkip to conversation

scsi: PR live migration PREEMPT fixes

PATCH Applied Stefan Hajnoczi 2 participants 8 messages Apr 1 – May 8 SCSI

lore
Share & tools
lore
#0 lore raw

cover scsi: PR live migration PREEMPT fixes

These patches are for QEMU 11.1 and 11.0-stable because they are not release blockers.

Testing SCSI Persistent Reservation live migration with the Linux LIO iSCSI target exposed two issues:

1. The LIO target interprets the request's TYPE field even though the spec says it should be ignored. This led to PREEMPT commands failing on the LIO target. Work around this by sending a "valid" TYPE field value. I have also contacted the LIO maintainers about improving spec-compliance but QEMU needs to work with existing targets.

2. The LIO target removes all matching registrations, including the I_T nexus that is sending the PREEMPT command, when PREEMPT is used to remove registrations rather than preempt a reservation. This actually appears to comply with the spec, but the hardware target I previously tested against would not remove the calling I_T nexus' registration and therefore QEMU's behavior was wrong. Implement a fix that works regardless of the target.

Stefan Hajnoczi (3):
scsi: adjust error_prepend() formatting
scsi: always send valid PREEMPT TYPE field
scsi: register again after PREEMPT without reservation

 include/scsi/constants.h | 10 ++++++++++
 hw/scsi/scsi-generic.c   | 24 ++++++++++++++++++++++--
 2 files changed, 32 insertions(+), 2 deletions(-)
Signature · 1 line
2.53.0
#1 lore raw

PATCH 1/3 scsi: adjust error_prepend() formatting

The error strings will be concatenated so add a separator to make the combined error message easy to read.

---
 hw/scsi/scsi-generic.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
hw/scsi/scsi-generic.c +2 -2
index a2316a5266..b55d068323 100644
@@ -439,7 +439,7 @@ static bool scsi_generic_pr_register(SCSIDevice *s, uint64_t key, Error **errp)
     ret = scsi_SG_IO(s->conf.blk, SG_DXFER_TO_DEV, cmd, sizeof(cmd),
                      buf, sizeof(buf), s->io_timeout, errp);
     if (ret < 0) {
-        error_prepend(errp, "PERSISTENT RESERVE OUT with REGISTER");
+        error_prepend(errp, "PERSISTENT RESERVE OUT with REGISTER: ");
         return false;
     }
     return true;
@@ -463,7 +463,7 @@ static bool scsi_generic_pr_preempt(SCSIDevice *s, uint64_t key,
     ret = scsi_SG_IO(s->conf.blk, SG_DXFER_TO_DEV, cmd, sizeof(cmd),
                      buf, sizeof(buf), s->io_timeout, errp);
     if (ret < 0) {
-        error_prepend(errp, "PERSISTENT RESERVE OUT with PREEMPT");
+        error_prepend(errp, "PERSISTENT RESERVE OUT with PREEMPT: ");
         return false;
     }
     return true;
Signature · 1 line
2.53.0
#2 lore raw

PATCH 2/3 scsi: always send valid PREEMPT TYPE field

In reply to Stefan Hajnoczi

The SPC-6 specification says that the PREEMPT service action ignores the TYPE field when there is no reservation. However, the LIO Linux iSCSI target rejects commands with a zero TYPE field. The field never ends up being used in this case, so replace it with a "valid" value to work around the issue.

---
 include/scsi/constants.h | 10 ++++++++++
 hw/scsi/scsi-generic.c   | 10 ++++++++++
 2 files changed, 20 insertions(+)
include/scsi/constants.h +10 -0
index cb97bdb636..717e470a5d 100644
@@ -340,4 +340,14 @@
 #define PRO_REGISTER_AND_MOVE                   0x07
 #define PRO_REPLACE_LOST_RESERVATION            0x08
 
+/*
+ * Persistent reservation types
+ */
+#define PR_TYPE_WRITE_EXCLUSIVE             0x1
+#define PR_TYPE_EXCLUSIVE_ACCESS            0x3
+#define PR_TYPE_WRITE_EXCLUSIVE_REG_ONLY    0x5
+#define PR_TYPE_EXCLUSIVE_ACCESS_REG_ONLY   0x6
+#define PR_TYPE_WRITE_EXCLUSIVE_ALL_REGS    0x7
+#define PR_TYPE_EXCLUSIVE_ACCESS_ALL_REGS   0x8
+
 #endif
hw/scsi/scsi-generic.c +10 -0
index b55d068323..349dea6bdd 100644
@@ -453,6 +453,16 @@ static bool scsi_generic_pr_preempt(SCSIDevice *s, uint64_t key,
     uint64_t key_be = cpu_to_be64(key);
     int ret;
 
+    /*
+     * The LIO iSCSI target in Linux up to at least version 7.0 rejects PREEMPT
+     * commands with a zero TYPE field although the SPC-6 specification says
+     * the field should be ignored when there is no persistent reservation.
+     * Work around this by choosing an arbitrary valid PR type value.
+     */
+    if (resv_type == 0) {
+        resv_type = PR_TYPE_WRITE_EXCLUSIVE;
+    }
+
     cmd[0] = PERSISTENT_RESERVE_OUT;
     cmd[1] = PRO_PREEMPT;
     cmd[2] = resv_type & 0xf;
Signature · 1 line
2.53.0
#3 lore raw

PATCH 3/3 scsi: register again after PREEMPT without reservation

In reply to Stefan Hajnoczi

The SCSI specification says PREEMPT without a reservation removes all registrations with the given key. Try to register again after PREEMPT since our key will have been removed.

In practice some SCSI targets keep the calling I_T nexus' registration instead of removing it. Therefore we need to handle both the spec-compliant and the non-compliant behavior.

---
 hw/scsi/scsi-generic.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
hw/scsi/scsi-generic.c +10 -0
index 349dea6bdd..5182f9236d 100644
@@ -513,6 +513,16 @@ bool scsi_generic_pr_state_preempt(SCSIDevice *s, Error **errp)
         if (!scsi_generic_pr_preempt(s, key, resv_type, errp)) {
             return false;
         }
+
+        /*
+         * Some SCSI targets, like the Linux LIO target, remove our
+         * registration when preempting without a reservation (resv_type is 0).
+         * Try to register again but ignore the error since a RESERVATION
+         * CONFLICT is expected if our registration remained in place.
+         */
+        if (resv_type == 0) {
+            scsi_generic_pr_register(s, key, NULL);
+        }
     }
     return true;
 }
Signature · 1 line
2.53.0
#4 lore raw

Re: [PATCH 3/3] scsi: register again after PREEMPT without reservation

Quoted 21 lines · Stefan Hajnoczi

The SCSI specification says PREEMPT without a reservation removes all registrations with the given key. Try to register again after PREEMPT since our key will have been removed.

In practice some SCSI targets keep the calling I_T nexus' registration instead of removing it. Therefore we need to handle both the spec-compliant and the non-compliant behavior.

---
  hw/scsi/scsi-generic.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
hw/scsi/scsi-generic.c +0 -0
index 349dea6bdd..5182f9236d 100644
@@ -513,6 +513,16 @@ bool scsi_generic_pr_state_preempt(SCSIDevice *s, Error **errp)
          if (!scsi_generic_pr_preempt(s, key, resv_type, errp)) {
              return false;
          }

hw/scsi/scsi-generic.c bool scsi_generic_pr_state_preempt(SCSIDevice *s, Error **errp) · line 525

Quoted 10 lines
hw/scsi/scsi-generic.c +10 -0
+
+        /*
+         * Some SCSI targets, like the Linux LIO target, remove our
+         * registration when preempting without a reservation (resv_type is 0).
+         * Try to register again but ignore the error since a RESERVATION
+         * CONFLICT is expected if our registration remained in place.
+         */
+        if (resv_type == 0) {
+            scsi_generic_pr_register(s, key, NULL);
+        }

Hi Stefan,

I'm replying here for both patch 2 and patch 3.

Can this happen after the previous patch? (Also it's conflicting to say
in patch 2 that LIO rejects PREEMPT, but also in this one that LIO
unregisters the key).

And for patch 2, is a RELEASE needed after issuing PREEMPT with an artificial PR_TYPE_WRITE_EXCLUSIVE?

Paolo

#5 lore raw

Re: [PATCH 3/3] scsi: register again after PREEMPT without reservation

Quoted 35 lines · Paolo Bonzini
Quoted 21 lines · Stefan Hajnoczi

The SCSI specification says PREEMPT without a reservation removes all registrations with the given key. Try to register again after PREEMPT since our key will have been removed.

In practice some SCSI targets keep the calling I_T nexus' registration instead of removing it. Therefore we need to handle both the spec-compliant and the non-compliant behavior.

---
  hw/scsi/scsi-generic.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
hw/scsi/scsi-generic.c +0 -0
index 349dea6bdd..5182f9236d 100644
@@ -513,6 +513,16 @@ bool scsi_generic_pr_state_preempt(SCSIDevice *s, Error **errp)
          if (!scsi_generic_pr_preempt(s, key, resv_type, errp)) {
              return false;
          }

hw/scsi/scsi-generic.c bool scsi_generic_pr_state_preempt(SCSIDevice *s, Error **errp) · line 525

Quoted 10 lines
hw/scsi/scsi-generic.c +10 -0
+
+        /*
+         * Some SCSI targets, like the Linux LIO target, remove our
+         * registration when preempting without a reservation (resv_type is 0).
+         * Try to register again but ignore the error since a RESERVATION
+         * CONFLICT is expected if our registration remained in place.
+         */
+        if (resv_type == 0) {
+            scsi_generic_pr_register(s, key, NULL);
+        }

Hi Stefan,

Quoted 5 lines

I'm replying here for both patch 2 and patch 3.

Can this happen after the previous patch? (Also it's conflicting to say in patch 2 that LIO rejects PREEMPT, but also in this one that LIO unregisters the key).

Yes. Both patches are needed.

The first patch gets LIO to process the PREEMPT. Before LIO would just fail the command.

But once LIO processes the PREEMPT, we hit the next problem: PREEMPT completes but our registration has been removed. So now we need to register again after PREEMPT.

Quoted 2 lines

And for patch 2, is a RELEASE needed after issuing PREEMPT with an artificial PR_TYPE_WRITE_EXCLUSIVE?

No, because PREEMPT will not acquire a reservation when there is no reservation holder with the given key. PREEMPT will only remove registrations in that case.

Stefan

Reviewed-by
#6 lore raw

Re: [PATCH 3/3] scsi: register again after PREEMPT without reservation

Quoted 7 lines · Stefan Hajnoczi

The first patch gets LIO to process the PREEMPT. Before LIO would just fail the command.

But once LIO processes the PREEMPT, we hit the next problem: PREEMPT completes but our registration has been removed. So now we need to register again after PREEMPT.

Quoted 6 lines
Quoted 2 lines

And for patch 2, is a RELEASE needed after issuing PREEMPT with an artificial PR_TYPE_WRITE_EXCLUSIVE?

No, because PREEMPT will not acquire a reservation when there is no reservation holder with the given key. PREEMPT will only remove registrations in that case.

Thanks for explaining all this. I didn't understand the behavior where the source does not have a reservation. Series

Paolo