Skip to contentSkip to conversation

hw/block/nvme: cmb enhancements and bump to v1.4

PATCH Superseded Klaus Jensen 2 participants 5 messages Dec 18, 2020

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#1 lore raw

PATCH 1/3 hw/block/nvme: cmb enhancements

In reply to Klaus Jensen

From: Padmakar Kalghatgi <[email protected]>

Implement v1.4 logic for configuring the Controller Memory Buffer. The x-legacy-cmb nvme device parameter can be set to revert to v1.3 behavior.

---
 hw/block/nvme.h      |   1 +
 include/block/nvme.h | 106 +++++++++++++++++++++++++++++++++++++++----
 hw/block/nvme.c      |  86 ++++++++++++++++++++++++++++-------
 3 files changed, 167 insertions(+), 26 deletions(-)
hw/block/nvme.h +1 -0
index 574333caa3f9..daeaa24eb2e4 100644
@@ -16,6 +16,7 @@ typedef struct NvmeParams {
     uint32_t aer_max_queued;
     uint8_t  mdts;
     bool     use_intel_id;
+    bool     legacy_cmb;
 } NvmeParams;
 
 typedef struct NvmeAsyncEvent {
include/block/nvme.h +96 -10

158 lines not on this page · raw message

hw/block/nvme.c +70 -16

129 lines not on this page · raw message

Signature · 1 line
2.29.2
#2 lore raw

PATCH 2/3 hw/block/nvme: bump to v1.4

In reply to Klaus Jensen

From: Klaus Jensen <[email protected]>

With the new CMB logic in place, bump the implemented specification version to v1.4.

---
 hw/block/nvme.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
hw/block/nvme.c +1 -1
index f3c111ee0a5c..16bf05638bf6 100644
@@ -74,7 +74,7 @@
 
 #define NVME_MAX_IOQPAIRS 0xffff
 #define NVME_DB_SIZE  4
-#define NVME_SPEC_VER 0x00010300
+#define NVME_SPEC_VER 0x00010400
 #define NVME_CMB_BIR 2
 #define NVME_PMR_BIR 2
 #define NVME_TEMPERATURE 0x143
Signature · 1 line
2.29.2
#3 lore raw

PATCH 3/3 hw/block/nvme: lift cmb restrictions

In reply to Klaus Jensen

From: Klaus Jensen <[email protected]>

The controller now implements v1.4 and we can lift the restrictions on CMB Data Pointer and Command Independent Locations Support (CDPCILS) and CMB Data Pointer Mixed Locations Support (CDPMLS) since the device really does not care about mixed host/cmb pointers in those cases.

---
 hw/block/nvme.c | 33 ++-------------------------------
 1 file changed, 2 insertions(+), 31 deletions(-)
hw/block/nvme.c +2 -31
index 16bf05638bf6..bd1de8453cfa 100644
@@ -336,7 +336,6 @@ static uint16_t nvme_map_prp(NvmeCtrl *n, uint64_t prp1, uint64_t prp2,
     trans_len = MIN(len, trans_len);
     int num_prps = (len >> n->page_bits) + 1;
     uint16_t status;
-    bool prp_list_in_cmb = false;
     int ret;
 
     QEMUSGList *qsg = &req->qsg;
@@ -362,10 +361,6 @@ static uint16_t nvme_map_prp(NvmeCtrl *n, uint64_t prp1, uint64_t prp2,
             uint32_t nents, prp_trans;
             int i = 0;
 
-            if (nvme_addr_is_cmb(n, prp2)) {
-                prp_list_in_cmb = true;
-            }
-
             nents = (len + n->page_size - 1) >> n->page_bits;
             prp_trans = MIN(n->max_prp_ents, nents) * sizeof(uint64_t);
             ret = nvme_addr_read(n, prp2, (void *)prp_list, prp_trans);
@@ -382,10 +377,6 @@ static uint16_t nvme_map_prp(NvmeCtrl *n, uint64_t prp1, uint64_t prp2,
                         return NVME_INVALID_PRP_OFFSET | NVME_DNR;
                     }
 
-                    if (prp_list_in_cmb != nvme_addr_is_cmb(n, prp_ent)) {
-                        return NVME_INVALID_USE_OF_CMB | NVME_DNR;
-                    }
-
                     i = 0;
                     nents = (len + n->page_size - 1) >> n->page_bits;
                     prp_trans = MIN(n->max_prp_ents, nents) * sizeof(uint64_t);
@@ -519,7 +510,6 @@ static uint16_t nvme_map_sgl(NvmeCtrl *n, QEMUSGList *qsg, QEMUIOVector *iov,
     uint64_t nsgld;
     uint32_t seg_len;
     uint16_t status;
-    bool sgl_in_cmb = false;
     hwaddr addr;
     int ret;
 
@@ -541,18 +531,6 @@ static uint16_t nvme_map_sgl(NvmeCtrl *n, QEMUSGList *qsg, QEMUIOVector *iov,
         goto out;
     }
 
-    /*
-     * If the segment is located in the CMB, the submission queue of the
-     * request must also reside there.
-     */
-    if (nvme_addr_is_cmb(n, addr)) {
-        if (!nvme_addr_is_cmb(n, req->sq->dma_addr)) {
-            return NVME_INVALID_USE_OF_CMB | NVME_DNR;
-        }
-
-        sgl_in_cmb = true;
-    }
-
     for (;;) {
         switch (NVME_SGL_TYPE(sgld->type)) {
         case NVME_SGL_DESCR_TYPE_SEGMENT:
@@ -641,15 +619,6 @@ static uint16_t nvme_map_sgl(NvmeCtrl *n, QEMUSGList *qsg, QEMUIOVector *iov,
         if (status) {
             goto unmap;
         }
-
-        /*
-         * If the next segment is in the CMB, make sure that the sgl was
-         * already located there.
-         */
-        if (sgl_in_cmb != nvme_addr_is_cmb(n, addr)) {
-            status = NVME_INVALID_USE_OF_CMB | NVME_DNR;
-            goto unmap;
-        }
     }
 
 out:
@@ -2428,6 +2397,8 @@ static int nvme_start_ctrl(NvmeCtrl *n)
 
 static void nvme_cmb_enable_regs(NvmeCtrl *n)
 {
+    NVME_CMBLOC_SET_CDPCILS(n->bar.cmbloc, 1);
+    NVME_CMBLOC_SET_CDPMLS(n->bar.cmbloc, 1);
     NVME_CMBLOC_SET_BIR(n->bar.cmbloc, NVME_CMB_BIR);
 
     NVME_CMBSZ_SET_SQS(n->bar.cmbsz, 1);
Signature · 1 line
2.29.2
#0 lore raw

cover hw/block/nvme: cmb enhancements and bump to v1.4

From: Klaus Jensen <[email protected]>

This adds CMB logic from v1.4. To my knowledge, this is the last piece missing for v1.4 compliance, so bump the controller spec version. Please retort if I am jumping the gun.

Since the slow-moving (sorry, super poor attempt of humor) Linux kernel nvme driver does not support v1.4 CMB, this series adds a 'x-legacy-cmb' nvme device parameter that reverts the CMB configuration behavior to v1.3, thus allowing the kernel to continue using the CMB for submission queues.

The question here is if we should rather skip that parameter, only support v1.4 behavior and point the finger at the kernel nvme gang. I'm leaning towards this.

Keith, what is your opinion on this?

Klaus Jensen (2):
hw/block/nvme: bump to v1.4
hw/block/nvme: lift cmb restrictions

Padmakar Kalghatgi (1):
hw/block/nvme: cmb enhancements

 hw/block/nvme.h      |   1 +
 include/block/nvme.h | 106 +++++++++++++++++++++++++++++++++----
 hw/block/nvme.c      | 121 ++++++++++++++++++++++++++-----------------
 3 files changed, 170 insertions(+), 58 deletions(-)
Signature · 1 line
2.29.2
#4 lore raw
Quoted 12 lines · Klaus Jensen

From: Klaus Jensen <[email protected]>

This adds CMB logic from v1.4. To my knowledge, this is the last piece missing for v1.4 compliance, so bump the controller spec version. Please retort if I am jumping the gun.

Since the slow-moving (sorry, super poor attempt of humor) Linux kernel nvme driver does not support v1.4 CMB, this series adds a 'x-legacy-cmb' nvme device parameter that reverts the CMB configuration behavior to v1.3, thus allowing the kernel to continue using the CMB for submission queues.

Quoted 5 lines

The question here is if we should rather skip that parameter, only support v1.4 behavior and point the finger at the kernel nvme gang. I'm leaning towards this.

Keith, what is your opinion on this?

The driver may know if it's running in a VM, but it doesn't necessarily know if its device is a passthrough vs. emulated. In the passthrough case, we can't use the CMB at all for <= 1.3 versions since the GPA vs HPA are not being considered. IMO, the kernel driver does need to be updated to suppress CMB usage for pre-1.4, and it should also add the CMBMSC capabilities for 1.4+.

Going forward with the qemu controller, I would remove legacy support for CMB entirely since every host should be able to interop with it without CMB support.