Skip to contentSkip to conversation

extend limit of physical sections number

PATCH In review Amos Kong 3 participants 8 messages Sep 27, 2013 – Nov 19, 2013

lore
Share & tools
lore

Amos Kong

author
#0 lore raw

PATCH 1/1 extend limit of physical sections number

 # qemu -drive file=/disk0,if=none,id=v0,format=qcow2 \
 -device virtio-blk-pci,drive=v0,id=v00,multifunction=on,addr=0x04.0
 ....

Launching guest with more than 32 virtio-blk disks, qemu will crash, because there are too many BARs.

This patch brings the limit of non-tcg up by a factor of 8 (32767 / 4096), i.e. 32*8 = 256.

---
 exec.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
exec.c +12 -5
index 5aef833..f639c01 100644
@@ -763,11 +763,18 @@ void phys_mem_set_alloc(void *(*alloc)(ram_addr_t))
 
 static uint16_t phys_section_add(MemoryRegionSection *section)
 {
-    /* The physical section number is ORed with a page-aligned
-     * pointer to produce the iotlb entries.  Thus it should
-     * never overflow into the page-aligned value.
-     */
-    assert(next_map.sections_nb < TARGET_PAGE_SIZE);
+    if (tcg_enabled()) {
+        /* The physical section number is ORed with a page-aligned
+         * pointer to produce the iotlb entries.  Thus it should
+         * never overflow into the page-aligned value.
+         */
+        assert(next_map.sections_nb < TARGET_PAGE_SIZE);
+    } else {
+        /* For KVM or Xen we can use the full range of the ptr field
+         * in PhysPageEntry.
+         */
+        assert(next_map.sections_nb < SHRT_MAX);
+    }
 
     if (next_map.sections_nb == next_map.sections_nb_alloc) {
         next_map.sections_nb_alloc = MAX(next_map.sections_nb_alloc * 2,
Signature · 1 line
1.8.3.1
#1 lore raw

Il 27/09/2013 18:49, Amos Kong ha scritto:

Quoted 35 lines
 # qemu -drive file=/disk0,if=none,id=v0,format=qcow2 \
 -device virtio-blk-pci,drive=v0,id=v00,multifunction=on,addr=0x04.0
 ....

Launching guest with more than 32 virtio-blk disks, qemu will crash, because there are too many BARs.

This patch brings the limit of non-tcg up by a factor of 8 (32767 / 4096), i.e. 32*8 = 256.

---
 exec.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
exec.c +5 -5
index 5aef833..f639c01 100644
@@ -763,11 +763,18 @@ void phys_mem_set_alloc(void *(*alloc)(ram_addr_t))
 
 static uint16_t phys_section_add(MemoryRegionSection *section)
 {
-    /* The physical section number is ORed with a page-aligned
-     * pointer to produce the iotlb entries.  Thus it should
-     * never overflow into the page-aligned value.
-     */
-    assert(next_map.sections_nb < TARGET_PAGE_SIZE);
+    if (tcg_enabled()) {
+        /* The physical section number is ORed with a page-aligned
+         * pointer to produce the iotlb entries.  Thus it should
+         * never overflow into the page-aligned value.
+         */

exec.c void phys_mem_set_alloc(void *(*alloc)(ram_addr_t)) · line 780

Quoted 10 lines
exec.c +7 -0
+        assert(next_map.sections_nb < TARGET_PAGE_SIZE);
+    } else {
+        /* For KVM or Xen we can use the full range of the ptr field
+         * in PhysPageEntry.
+         */
+        assert(next_map.sections_nb < SHRT_MAX);
+    }
 
     if (next_map.sections_nb == next_map.sections_nb_alloc) {
         next_map.sections_nb_alloc = MAX(next_map.sections_nb_alloc * 2,

Thanks for testing this patch Amos!

Paolo

author
#2 lore raw
Quoted 13 lines · Paolo Bonzini

Il 27/09/2013 18:49, Amos Kong ha scritto:

Quoted 12 lines
 # qemu -drive file=/disk0,if=none,id=v0,format=qcow2 \
 -device virtio-blk-pci,drive=v0,id=v00,multifunction=on,addr=0x04.0
 ....

Launching guest with more than 32 virtio-blk disks, qemu will crash, because there are too many BARs.

This patch brings the limit of non-tcg up by a factor of 8 (32767 / 4096), i.e. 32*8 = 256.

Hi Anthony, others

Can you help to review this patch?

Thanks, Amos

Quoted 37 lines
Quoted 23 lines
---
 exec.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
exec.c +5 -5
index 5aef833..f639c01 100644
@@ -763,11 +763,18 @@ void phys_mem_set_alloc(void *(*alloc)(ram_addr_t))
 
 static uint16_t phys_section_add(MemoryRegionSection *section)
 {
-    /* The physical section number is ORed with a page-aligned
-     * pointer to produce the iotlb entries.  Thus it should
-     * never overflow into the page-aligned value.
-     */
-    assert(next_map.sections_nb < TARGET_PAGE_SIZE);
+    if (tcg_enabled()) {
+        /* The physical section number is ORed with a page-aligned
+         * pointer to produce the iotlb entries.  Thus it should
+         * never overflow into the page-aligned value.
+         */

exec.c void phys_mem_set_alloc(void *(*alloc)(ram_addr_t)) · line 780

Quoted 10 lines
exec.c +7 -0
+        assert(next_map.sections_nb < TARGET_PAGE_SIZE);
+    } else {
+        /* For KVM or Xen we can use the full range of the ptr field
+         * in PhysPageEntry.
+         */
+        assert(next_map.sections_nb < SHRT_MAX);
+    }
 
     if (next_map.sections_nb == next_map.sections_nb_alloc) {
         next_map.sections_nb_alloc = MAX(next_map.sections_nb_alloc * 2,

Thanks for testing this patch Amos!

Paolo

#3 lore raw
In reply to Amos Kong
Quoted 31 lines · Amos Kong
 # qemu -drive file=/disk0,if=none,id=v0,format=qcow2 \
 -device virtio-blk-pci,drive=v0,id=v00,multifunction=on,addr=0x04.0
 ....

Launching guest with more than 32 virtio-blk disks, qemu will crash, because there are too many BARs.

This patch brings the limit of non-tcg up by a factor of 8 (32767 / 4096), i.e. 32*8 = 256.

---
 exec.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
exec.c +2 -5
index 5aef833..f639c01 100644
@@ -763,11 +763,18 @@ void phys_mem_set_alloc(void *(*alloc)(ram_addr_t))
 
 static uint16_t phys_section_add(MemoryRegionSection *section)
 {
-    /* The physical section number is ORed with a page-aligned
-     * pointer to produce the iotlb entries.  Thus it should
-     * never overflow into the page-aligned value.
-     */
-    assert(next_map.sections_nb < TARGET_PAGE_SIZE);
+    if (tcg_enabled()) {
+        /* The physical section number is ORed with a page-aligned

exec.c void phys_mem_set_alloc(void *(*alloc)(ram_addr_t)) · line 777

Quoted 10 lines
exec.c +10 -0
+         * pointer to produce the iotlb entries.  Thus it should
+         * never overflow into the page-aligned value.
+         */
+        assert(next_map.sections_nb < TARGET_PAGE_SIZE);
+    } else {
+        /* For KVM or Xen we can use the full range of the ptr field
+         * in PhysPageEntry.
+         */
+        assert(next_map.sections_nb < SHRT_MAX);
+    }

This looks really weird. Why should the memory subsystem care whether we're using TCG or KVM or Xen?

-- PMM

#4 lore raw

Il 05/11/2013 01:36, Peter Maydell ha scritto:

Quoted 37 lines · Amos Kong
Quoted 36 lines
 # qemu -drive file=/disk0,if=none,id=v0,format=qcow2 \
 -device virtio-blk-pci,drive=v0,id=v00,multifunction=on,addr=0x04.0
 ....

Launching guest with more than 32 virtio-blk disks, qemu will crash, because there are too many BARs.

This patch brings the limit of non-tcg up by a factor of 8 (32767 / 4096), i.e. 32*8 = 256.

---
 exec.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
exec.c +7 -5
index 5aef833..f639c01 100644
@@ -763,11 +763,18 @@ void phys_mem_set_alloc(void *(*alloc)(ram_addr_t))
 
 static uint16_t phys_section_add(MemoryRegionSection *section)
 {
-    /* The physical section number is ORed with a page-aligned
-     * pointer to produce the iotlb entries.  Thus it should
-     * never overflow into the page-aligned value.
-     */
-    assert(next_map.sections_nb < TARGET_PAGE_SIZE);
+    if (tcg_enabled()) {
+        /* The physical section number is ORed with a page-aligned
+         * pointer to produce the iotlb entries.  Thus it should
+         * never overflow into the page-aligned value.
+         */
+        assert(next_map.sections_nb < TARGET_PAGE_SIZE);
+    } else {
Quoted 8 lines

exec.c void phys_mem_set_alloc(void *(*alloc)(ram_addr_t)) · line 777

Quoted 5 lines
exec.c +5 -0
+        /* For KVM or Xen we can use the full range of the ptr field
+         * in PhysPageEntry.
+         */
+        assert(next_map.sections_nb < SHRT_MAX);
+    }

This looks really weird. Why should the memory subsystem care whether we're using TCG or KVM or Xen?

Because only TCG stores the section number in the low bits of the iotlb entry. This is exactly what is explained in the comments.

Paolo

#5 lore raw
Quoted 9 lines · Paolo Bonzini

Il 05/11/2013 01:36, Peter Maydell ha scritto:

Quoted 8 lines · Amos Kong
Quoted 7 lines
 # qemu -drive file=/disk0,if=none,id=v0,format=qcow2 \
 -device virtio-blk-pci,drive=v0,id=v00,multifunction=on,addr=0x04.0
 ....

Launching guest with more than 32 virtio-blk disks, qemu will crash, because there are too many BARs.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

This patch brings the limit of non-tcg up by a factor of 8 (32767 / 4096), i.e. 32*8 = 256.

This looks really weird. Why should the memory subsystem care whether we're using TCG or KVM or Xen?

Because only TCG stores the section number in the low bits of the iotlb entry. This is exactly what is explained in the comments.

So presumably we still crash if there are more than 32 virtio-blk disks on TCG (and indeed if more than 256 on KVM)? That doesn't seem very satisfactory...

-- PMM

#6 lore raw

Il 05/11/2013 13:23, Peter Maydell ha scritto:

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

This looks really weird. Why should the memory subsystem care whether we're using TCG or KVM or Xen?

Because only TCG stores the section number in the low bits of the iotlb entry. This is exactly what is explained in the comments.

So presumably we still crash if there are more than 32 virtio-blk disks on TCG (and indeed if more than 256 on KVM)? That doesn't seem very satisfactory...

It isn't, do you have any idea on how to make the threshold equal for TCG and KVM?

I guess the code could be made clearer like this:

    assert (... < SHRT_MAX);
    if (tcg_enabled()) {
        /* TCG has a stricter limit due to iotlb etc. etc. */
        assert (... < TARGET_PAGE_SIZE);
    }

but that's pretty much it...

Paolo

author
#7 lore raw
In reply to Amos Kong
Quoted 9 lines · Amos Kong
 # qemu -drive file=/disk0,if=none,id=v0,format=qcow2 \
 -device virtio-blk-pci,drive=v0,id=v00,multifunction=on,addr=0x04.0
 ....

Launching guest with more than 32 virtio-blk disks, qemu will crash, because there are too many BARs.

This patch brings the limit of non-tcg up by a factor of 8 (32767 / 4096), i.e. 32*8 = 256.

I re-tested with latest guest kernel (3.10.0-rc5), crash still occurred after applied this patch.

1. phys_section_add: assert(next_map.sections_nb < TARGET_PAGE_SIZE); crash

2. phys_section_add: assert(next_map.sections_nb < SHRT_MAX); crash

3. without this assert of next_map.sections_nb in phys_section_add crash occurred at register_subpage():
assert(existing->mr->subpage || existing->mr == &io_mem_unassigned);

Quoted 37 lines
---
 exec.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
exec.c +12 -5
index 5aef833..f639c01 100644
@@ -763,11 +763,18 @@ void phys_mem_set_alloc(void *(*alloc)(ram_addr_t))
 
 static uint16_t phys_section_add(MemoryRegionSection *section)
 {
-    /* The physical section number is ORed with a page-aligned
-     * pointer to produce the iotlb entries.  Thus it should
-     * never overflow into the page-aligned value.
-     */
-    assert(next_map.sections_nb < TARGET_PAGE_SIZE);
+    if (tcg_enabled()) {
+        /* The physical section number is ORed with a page-aligned
+         * pointer to produce the iotlb entries.  Thus it should
+         * never overflow into the page-aligned value.
+         */
+        assert(next_map.sections_nb < TARGET_PAGE_SIZE);
+    } else {
+        /* For KVM or Xen we can use the full range of the ptr field
+         * in PhysPageEntry.
+         */
+        assert(next_map.sections_nb < SHRT_MAX);
+    }
 
     if (next_map.sections_nb == next_map.sections_nb_alloc) {
         next_map.sections_nb_alloc = MAX(next_map.sections_nb_alloc * 2,
Signature · 1 line
1.8.3.1
Signature · 1 line
			Amos.