Skip to contentSkip to conversation

hw/nvme: fix mmio read

PATCH Superseded Klaus Jensen 3 participants 7 messages Jul 13, 2021

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 hw/nvme: fix mmio read

From: Klaus Jensen <[email protected]>

The new PMR test unearthed a long-standing issue with MMIO reads on big-endian hosts.

Fix by using the ldn_he_p helper instead of memcpy.

---
 hw/nvme/ctrl.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
hw/nvme/ctrl.c +7 -7
index 2f0524e12a36..dd81c3b19c7e 100644
@@ -5951,7 +5951,6 @@ static uint64_t nvme_mmio_read(void *opaque, hwaddr addr, unsigned size)
 {
     NvmeCtrl *n = (NvmeCtrl *)opaque;
     uint8_t *ptr = (uint8_t *)&n->bar;
-    uint64_t val = 0;
 
     trace_pci_nvme_mmio_read(addr, size);
 
@@ -5977,14 +5976,15 @@ static uint64_t nvme_mmio_read(void *opaque, hwaddr addr, unsigned size)
             (NVME_PMRCAP_PMRWBM(n->bar.pmrcap) & 0x02)) {
             memory_region_msync(&n->pmr.dev->mr, 0, n->pmr.dev->size);
         }
-        memcpy(&val, ptr + addr, size);
-    } else {
-        NVME_GUEST_ERR(pci_nvme_ub_mmiord_invalid_ofs,
-                       "MMIO read beyond last register,"
-                       " offset=0x%"PRIx64", returning 0", addr);
+
+        return ldn_he_p(ptr + addr, size);
     }
 
-    return val;
+    NVME_GUEST_ERR(pci_nvme_ub_mmiord_invalid_ofs,
+                   "MMIO read beyond last register,"
+                   " offset=0x%"PRIx64", returning 0", addr);
+
+    return 0;
 }
 
 static void nvme_process_db(NvmeCtrl *n, hwaddr addr, int val)
Signature · 1 line
2.32.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 52 lines · Klaus Jensen

From: Klaus Jensen <[email protected]>

The new PMR test unearthed a long-standing issue with MMIO reads on big-endian hosts.

Fix by using the ldn_he_p helper instead of memcpy.

---
hw/nvme/ctrl.c | 14 +++++++-------

1 file changed, 7 insertions(+), 7 deletions(-)
hw/nvme/ctrl.c +7 -7
index 2f0524e12a36..dd81c3b19c7e 100644
@@ -5951,7 +5951,6 @@ static uint64_t nvme_mmio_read(void *opaque, hwaddr addr, unsigned size)
 {
    NvmeCtrl *n = (NvmeCtrl *)opaque;
    uint8_t *ptr = (uint8_t *)&n->bar;
-    uint64_t val = 0;
 
    trace_pci_nvme_mmio_read(addr, size);
 
@@ -5977,14 +5976,15 @@ static uint64_t nvme_mmio_read(void *opaque, hwaddr addr, unsigned size)
            (NVME_PMRCAP_PMRWBM(n->bar.pmrcap) & 0x02)) {
            memory_region_msync(&n->pmr.dev->mr, 0, n->pmr.dev->size);
        }
-        memcpy(&val, ptr + addr, size);
-    } else {
-        NVME_GUEST_ERR(pci_nvme_ub_mmiord_invalid_ofs,
-                       "MMIO read beyond last register,"
-                       " offset=0x%"PRIx64", returning 0", addr);
+
+        return ldn_he_p(ptr + addr, size);
    }
 
-    return val;
+    NVME_GUEST_ERR(pci_nvme_ub_mmiord_invalid_ofs,
+                   "MMIO read beyond last register,"
+                   " offset=0x%"PRIx64", returning 0", addr);
+
+    return 0;
 }
 
 static void nvme_process_db(NvmeCtrl *n, hwaddr addr, int val)
Signature · 1 line
2.32.0

LGTM.

#2 lore raw
In reply to Klaus Jensen
Quoted 28 lines · Klaus Jensen

From: Klaus Jensen <[email protected]>

The new PMR test unearthed a long-standing issue with MMIO reads on big-endian hosts.

Fix by using the ldn_he_p helper instead of memcpy.

---
 hw/nvme/ctrl.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
hw/nvme/ctrl.c +0 -1
index 2f0524e12a36..dd81c3b19c7e 100644
@@ -5951,7 +5951,6 @@ static uint64_t nvme_mmio_read(void *opaque, hwaddr addr, unsigned size)
 {
     NvmeCtrl *n = (NvmeCtrl *)opaque;
     uint8_t *ptr = (uint8_t *)&n->bar;
-    uint64_t val = 0;
 
     trace_pci_nvme_mmio_read(addr, size);
 

hw/nvme/ctrl.c static uint64_t nvme_mmio_read(void *opaque, hwaddr addr, unsigned size) · line 5980

Quoted 10 lines
hw/nvme/ctrl.c +2 -5
@@ -5977,14 +5976,15 @@ static uint64_t nvme_mmio_read(void *opaque, hwaddr addr, unsigned size)
             (NVME_PMRCAP_PMRWBM(n->bar.pmrcap) & 0x02)) {
             memory_region_msync(&n->pmr.dev->mr, 0, n->pmr.dev->size);
         }
-        memcpy(&val, ptr + addr, size);
-    } else {
-        NVME_GUEST_ERR(pci_nvme_ub_mmiord_invalid_ofs,
-                       "MMIO read beyond last register,"
-                       " offset=0x%"PRIx64", returning 0", addr);
+
+        return ldn_he_p(ptr + addr, size);

I don't think this will do the right thing for accesses which aren't of the same width as whatever the field in NvmeBar is defined as. For instance, if the guest does a 32-bit access to offset 0, because the first field is defined as 'uint64_t cap', on a big-endian host they will end up reading the high 4 bytes of the 64-bit value, and on a little-endian host they will get the low 4 bytes.

hw/nvme/ctrl.c static uint64_t nvme_mmio_read(void *opaque, hwaddr addr, unsigned size) · line 5988

Quoted 9 lines
hw/nvme/ctrl.c +5 -1
     }
 
-    return val;
+    NVME_GUEST_ERR(pci_nvme_ub_mmiord_invalid_ofs,
+                   "MMIO read beyond last register,"
+                   " offset=0x%"PRIx64", returning 0", addr);
+
+    return 0;
 }

Looking at the surrounding code, I notice that we guard this "read size bytes from &n->bar + addr" with

    if (addr < sizeof(n->bar)) {

but that doesn't account for 'size', so if the guest asks to read 4 bytes starting at offset sizeof(n->bar)-1 then we'll still read 3 bytes beyond the end of the buffer...

thanks
-- PMM

#3 lore raw
Quoted 38 lines · Peter Maydell
Quoted 37 lines · Klaus Jensen

From: Klaus Jensen <[email protected]>

The new PMR test unearthed a long-standing issue with MMIO reads on big-endian hosts.

Fix by using the ldn_he_p helper instead of memcpy.

---
 hw/nvme/ctrl.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
hw/nvme/ctrl.c +1 -6
index 2f0524e12a36..dd81c3b19c7e 100644
@@ -5951,7 +5951,6 @@ static uint64_t nvme_mmio_read(void *opaque, hwaddr addr, unsigned size)
 {
     NvmeCtrl *n = (NvmeCtrl *)opaque;
     uint8_t *ptr = (uint8_t *)&n->bar;
-    uint64_t val = 0;
 
     trace_pci_nvme_mmio_read(addr, size);
 
@@ -5977,14 +5976,15 @@ static uint64_t nvme_mmio_read(void *opaque, hwaddr addr, unsigned size)
             (NVME_PMRCAP_PMRWBM(n->bar.pmrcap) & 0x02)) {
             memory_region_msync(&n->pmr.dev->mr, 0, n->pmr.dev->size);
         }
-        memcpy(&val, ptr + addr, size);
-    } else {
-        NVME_GUEST_ERR(pci_nvme_ub_mmiord_invalid_ofs,
-                       "MMIO read beyond last register,"
-                       " offset=0x%"PRIx64", returning 0", addr);
+
Quoted 9 lines

hw/nvme/ctrl.c static uint64_t nvme_mmio_read(void *opaque, hwaddr addr, unsigned size) · line 5980

Quoted 1 line
hw/nvme/ctrl.c +1 -0
+        return ldn_he_p(ptr + addr, size);

I don't think this will do the right thing for accesses which aren't of the same width as whatever the field in NvmeBar is defined as. For instance, if the guest does a 32-bit access to offset 0, because the first field is defined as 'uint64_t cap', on a big-endian host they will end up reading the high 4 bytes of the 64-bit value, and on a little-endian host they will get the low 4 bytes.

Thanks for taking a look Peter, I wondered if I actually fixed it correctly or not, and I obviously didnt.

I guess I can't get around handling 64 bit registers explicitly and convert them to little endian explicitly then.

Quoted 10 lines

hw/nvme/ctrl.c static uint64_t nvme_mmio_read(void *opaque, hwaddr addr, unsigned size) · line 5988

Quoted 9 lines
hw/nvme/ctrl.c +5 -1
     }
 
-    return val;
+    NVME_GUEST_ERR(pci_nvme_ub_mmiord_invalid_ofs,
+                   "MMIO read beyond last register,"
+                   " offset=0x%"PRIx64", returning 0", addr);
+
+    return 0;
 }
Quoted 7 lines

Looking at the surrounding code, I notice that we guard this "read size bytes from &n->bar + addr" with

    if (addr < sizeof(n->bar)) {

but that doesn't account for 'size', so if the guest asks to read 4 bytes starting at offset sizeof(n->bar)-1 then we'll still read 3 bytes beyond the end of the buffer...

The buffer is at least sizeof(n->bar) + 8 bytes (there are two doorbell registers following the controller registers). It is wrong for the host to read those, but as per the spec it is undefined behavior. I did consider reversing the condition to (addr > sizeof(n->bar) - size). I guess that would be the proper thing to do.

#4 lore raw
Quoted 6 lines · Klaus Jensen
Quoted 4 lines · Peter Maydell

Looking at the surrounding code, I notice that we guard this "read size bytes from &n->bar + addr" with

    if (addr < sizeof(n->bar)) {
Quoted 7 lines
Quoted 3 lines

but that doesn't account for 'size', so if the guest asks to read 4 bytes starting at offset sizeof(n->bar)-1 then we'll still read 3 bytes beyond the end of the buffer...

The buffer is at least sizeof(n->bar) + 8 bytes (there are two doorbell registers following the controller registers). It is wrong for the host to read those, but as per the spec it is undefined behavior.

I don't know about the doorbell registers, but with this code (or the old memcpy()) you'll access whatever the next thing after "NvmeBar bar" in the NvmeCtrl struct is, which looks like it's the first part of 'struct NvmeParams".

thanks
-- PMM

#5 lore raw
Quoted 11 lines · Peter Maydell
Quoted 10 lines · Klaus Jensen
Quoted 7 lines · Peter Maydell

Looking at the surrounding code, I notice that we guard this "read size bytes from &n->bar + addr" with

    if (addr < sizeof(n->bar)) {

but that doesn't account for 'size', so if the guest asks to read 4 bytes starting at offset sizeof(n->bar)-1 then we'll still read 3 bytes beyond the end of the buffer...

Quoted 9 lines
Quoted 3 lines

The buffer is at least sizeof(n->bar) + 8 bytes (there are two doorbell registers following the controller registers). It is wrong for the host to read those, but as per the spec it is undefined behavior.

I don't know about the doorbell registers, but with this code (or the old memcpy()) you'll access whatever the next thing after "NvmeBar bar" in the NvmeCtrl struct is, which looks like it's the first part of 'struct NvmeParams".

Sorry, you are of course right. I remembered how the bar was allocated incorrectly.

#6 lore raw
Quoted 16 lines · Klaus Jensen
Quoted 15 lines · Peter Maydell
Quoted 13 lines · Klaus Jensen
Quoted 7 lines · Peter Maydell

Looking at the surrounding code, I notice that we guard this "read size bytes from &n->bar + addr" with

    if (addr < sizeof(n->bar)) {

but that doesn't account for 'size', so if the guest asks to read 4 bytes starting at offset sizeof(n->bar)-1 then we'll still read 3 bytes beyond the end of the buffer...

The buffer is at least sizeof(n->bar) + 8 bytes (there are two doorbell registers following the controller registers). It is wrong for the host to read those, but as per the spec it is undefined behavior.

Quoted 8 lines
Quoted 5 lines

I don't know about the doorbell registers, but with this code (or the old memcpy()) you'll access whatever the next thing after "NvmeBar bar" in the NvmeCtrl struct is, which looks like it's the first part of 'struct NvmeParams".

Sorry, you are of course right. I remembered how the bar was allocated incorrectly.

I fixed this properly by holding all bar values in little endian as per the spec.

I'll clean it up and send it tonight.