Skip to contentSkip to conversation

spapr: Abort when HTAB size requirement can't be met

RFC PATCH In review Bharata B Rao 2 participants 9 messages Sep 22, 2015 – Sep 24, 2015

lore
Share & tools
lore
#0 lore raw

cover spapr: Abort when HTAB size requirement can't be met

HTAB size is a factor of maximum memory size that is specified by maxmem= command line option. In cases where there is shortage of host memory, host will not be able to allocate contiguous memory for guest HTAB and will instead allocate a smaller HTAB. This usually is not a problem but when user starts hotplugging memory to the guest, we can run out of HTAB entries and hence memory hotplug fails. This failure should have been handled gracefully by the guest kernel, but currently it leads to guest kernel OOPS. This will eventually get fixed when the handling of memory hotplug is completely moved to kernel for PowerKVM.

Prevent such kernel failure by refusing to boot the guest when requested HTAB size can't be allocated. However HTAB allocation happens in the reset path from where it is too late to abort. Hence this patchset moves the HTAB allocation to machine init and aborts if HTAB size requirement isn't met.

Two previous related attempts to address this problem:

1. http://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg03325.html
Aborts the guest from the reset path - not desirable.
2. https://lists.gnu.org/archive/html/qemu-devel/2015-08/msg02602.html
Silently disables memory hotplug - not desirable.

This patchset applies against David Gibson's spapr-next.

Bharata B Rao (2):
spapr: Allocate HTAB from machine init
spapr: Abort when HTAB of requested size isn't allocated

 hw/ppc/spapr.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)
Signature · 1 line
2.1.0
#1 lore raw

RFC PATCH 1/2 spapr: Allocate HTAB from machine init

Allocate HTAB from ppc_spapr_init() so that we can abort the guest if requested HTAB size is't allocated by the host. However retain the htab reset call in spapr_reset_htab() so that HTAB gets reset (and not allocated) during machine reset.

---
 hw/ppc/spapr.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
hw/ppc/spapr.c +12 -1
index 7f4f196..4692122 100644
@@ -979,7 +979,7 @@ static void emulate_spapr_hypercall(PowerPCCPU *cpu)
 #define CLEAN_HPTE(_hpte)  ((*(uint64_t *)(_hpte)) &= tswap64(~HPTE64_V_HPTE_DIRTY))
 #define DIRTY_HPTE(_hpte)  ((*(uint64_t *)(_hpte)) |= tswap64(HPTE64_V_HPTE_DIRTY))
 
-static void spapr_reset_htab(sPAPRMachineState *spapr)
+static void spapr_alloc_htab(sPAPRMachineState *spapr)
 {
     long shift;
     int index;
@@ -1012,6 +1012,16 @@ static void spapr_reset_htab(sPAPRMachineState *spapr)
             DIRTY_HPTE(HPTE(spapr->htab, index));
         }
     }
+}
+
+static void spapr_reset_htab(sPAPRMachineState *spapr)
+{
+    /*
+     * We have already allocated the hash page table, this call will
+     * not again allocate but only result in clearing of hash page
+     * table entries.
+     */
+    kvmppc_reset_htab(spapr->htab_shift);
 
     /* Update the RMA size if necessary */
     if (spapr->vrma_adjust) {
@@ -1709,6 +1719,7 @@ static void ppc_spapr_init(MachineState *machine)
         }
         spapr->htab_shift++;
     }
+    spapr_alloc_htab(spapr);
 
     /* Set up Interrupt Controller before we create the VCPUs */
     spapr->icp = xics_system_init(machine,
Signature · 1 line
2.1.0
#2 lore raw

Re: [Qemu-devel] [RFC PATCH v0 1/2] spapr: Allocate HTAB from machine init

Quoted 4 lines · Bharata B Rao

Allocate HTAB from ppc_spapr_init() so that we can abort the guest if requested HTAB size is't allocated by the host. However retain the htab reset call in spapr_reset_htab() so that HTAB gets reset (and not allocated) during machine reset.

I was briefly worried about this, because I recall there as a reason htab allocation got moved to the reset handler in the first place. Looking at the git history, however, I've convinced myself this is basically ok (because you preserve the call during reset to wipe clean the htab).

Quoted 22 lines
---
 hw/ppc/spapr.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
hw/ppc/spapr.c +1 -1
index 7f4f196..4692122 100644
@@ -979,7 +979,7 @@ static void emulate_spapr_hypercall(PowerPCCPU *cpu)
 #define CLEAN_HPTE(_hpte)  ((*(uint64_t *)(_hpte)) &= tswap64(~HPTE64_V_HPTE_DIRTY))
 #define DIRTY_HPTE(_hpte)  ((*(uint64_t *)(_hpte)) |= tswap64(HPTE64_V_HPTE_DIRTY))
 
-static void spapr_reset_htab(sPAPRMachineState *spapr)
+static void spapr_alloc_htab(sPAPRMachineState *spapr)
 {
     long shift;
     int index;
@@ -1012,6 +1012,16 @@ static void spapr_reset_htab(sPAPRMachineState *spapr)
             DIRTY_HPTE(HPTE(spapr->htab, index));
         }
     }

hw/ppc/spapr.c static void spapr_reset_htab(sPAPRMachineState *spapr) · line 1024

Quoted 10 lines
hw/ppc/spapr.c +10 -0
+}
+
+static void spapr_reset_htab(sPAPRMachineState *spapr)
+{
+    /*
+     * We have already allocated the hash page table, this call will
+     * not again allocate but only result in clearing of hash page
+     * table entries.
+     */
+    kvmppc_reset_htab(spapr->htab_shift);

It's unlikely the kernel will give us less htab than we already have, but we really should at least check for that. Probably not much we can do except abort() but at least we can give a useful error message.

Quoted 10 lines
hw/ppc/spapr.c +1 -0
     /* Update the RMA size if necessary */
     if (spapr->vrma_adjust) {
@@ -1709,6 +1719,7 @@ static void ppc_spapr_init(MachineState *machine)
         }
         spapr->htab_shift++;
     }
+    spapr_alloc_htab(spapr);
 
     /* Set up Interrupt Controller before we create the VCPUs */
     spapr->icp = xics_system_init(machine,
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#3 lore raw

Re: [Qemu-devel] [RFC PATCH v0 1/2] spapr: Allocate HTAB from machine init

Quoted 40 lines · David Gibson
Quoted 4 lines · Bharata B Rao

Allocate HTAB from ppc_spapr_init() so that we can abort the guest if requested HTAB size is't allocated by the host. However retain the htab reset call in spapr_reset_htab() so that HTAB gets reset (and not allocated) during machine reset.

I was briefly worried about this, because I recall there as a reason htab allocation got moved to the reset handler in the first place. Looking at the git history, however, I've convinced myself this is basically ok (because you preserve the call during reset to wipe clean the htab).

Quoted 28 lines
---
 hw/ppc/spapr.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
hw/ppc/spapr.c +7 -1
index 7f4f196..4692122 100644
@@ -979,7 +979,7 @@ static void emulate_spapr_hypercall(PowerPCCPU *cpu)
 #define CLEAN_HPTE(_hpte)  ((*(uint64_t *)(_hpte)) &= tswap64(~HPTE64_V_HPTE_DIRTY))
 #define DIRTY_HPTE(_hpte)  ((*(uint64_t *)(_hpte)) |= tswap64(HPTE64_V_HPTE_DIRTY))
 
-static void spapr_reset_htab(sPAPRMachineState *spapr)
+static void spapr_alloc_htab(sPAPRMachineState *spapr)
 {
     long shift;
     int index;
@@ -1012,6 +1012,16 @@ static void spapr_reset_htab(sPAPRMachineState *spapr)
             DIRTY_HPTE(HPTE(spapr->htab, index));
         }
     }
+}
+
+static void spapr_reset_htab(sPAPRMachineState *spapr)
+{
+    /*
+     * We have already allocated the hash page table, this call will
Quoted 8 lines

hw/ppc/spapr.c static void spapr_reset_htab(sPAPRMachineState *spapr) · line 1024

Quoted 4 lines
hw/ppc/spapr.c +4 -0
+     * not again allocate but only result in clearing of hash page
+     * table entries.
+     */
+    kvmppc_reset_htab(spapr->htab_shift);

It's unlikely the kernel will give us less htab than we already have, but we really should at least check for that. Probably not much we can do except abort() but at least we can give a useful error message.

With the change I am doing here, this is no longer an allocation path. Host kernel will just clear the HTAB and return the same htab_shift that we passed here. So do you think it still makes sense to check return value ?

Regards,
Bharata.

#4 lore raw

Re: [Qemu-devel] [RFC PATCH v0 1/2] spapr: Allocate HTAB from machine init

Quoted 50 lines · Bharata B Rao
Quoted 40 lines · David Gibson
Quoted 4 lines · Bharata B Rao

Allocate HTAB from ppc_spapr_init() so that we can abort the guest if requested HTAB size is't allocated by the host. However retain the htab reset call in spapr_reset_htab() so that HTAB gets reset (and not allocated) during machine reset.

I was briefly worried about this, because I recall there as a reason htab allocation got moved to the reset handler in the first place. Looking at the git history, however, I've convinced myself this is basically ok (because you preserve the call during reset to wipe clean the htab).

Quoted 28 lines
---
 hw/ppc/spapr.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
hw/ppc/spapr.c +7 -1
index 7f4f196..4692122 100644
@@ -979,7 +979,7 @@ static void emulate_spapr_hypercall(PowerPCCPU *cpu)
 #define CLEAN_HPTE(_hpte)  ((*(uint64_t *)(_hpte)) &= tswap64(~HPTE64_V_HPTE_DIRTY))
 #define DIRTY_HPTE(_hpte)  ((*(uint64_t *)(_hpte)) |= tswap64(HPTE64_V_HPTE_DIRTY))
 
-static void spapr_reset_htab(sPAPRMachineState *spapr)
+static void spapr_alloc_htab(sPAPRMachineState *spapr)
 {
     long shift;
     int index;
@@ -1012,6 +1012,16 @@ static void spapr_reset_htab(sPAPRMachineState *spapr)
             DIRTY_HPTE(HPTE(spapr->htab, index));
         }
     }
+}
+
+static void spapr_reset_htab(sPAPRMachineState *spapr)
+{
+    /*
+     * We have already allocated the hash page table, this call will
Quoted 8 lines

hw/ppc/spapr.c static void spapr_reset_htab(sPAPRMachineState *spapr) · line 1024

Quoted 4 lines
hw/ppc/spapr.c +4 -0
+     * not again allocate but only result in clearing of hash page
+     * table entries.
+     */
+    kvmppc_reset_htab(spapr->htab_shift);

It's unlikely the kernel will give us less htab than we already have, but we really should at least check for that. Probably not much we can do except abort() but at least we can give a useful error message.

Quoted 4 lines

With the change I am doing here, this is no longer an allocation path. Host kernel will just clear the HTAB and return the same htab_shift that we passed here. So do you think it still makes sense to check return value ?

That's the current kernel behaviour, but the interface doesn't guarantee that. So, yes, I still think you have to check the return value.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#5 lore raw

RFC PATCH 2/2 spapr: Abort when HTAB of requested size isn't allocated

In reply to Bharata B Rao

Terminate the guest when HTAB of requested size isn't allocated by the host.

When memory hotplug is attempted on a guest that has booted with less than requested HTAB size, the guest kernel will not be able to gracefully fail the hotplug request. This patch will ensure that we never end up in a situation where memory hotplug fails due to less than requested HTAB size.

---
 hw/ppc/spapr.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/ppc/spapr.c +4 -0
index 4692122..66446af 100644
@@ -992,6 +992,10 @@ static void spapr_alloc_htab(sPAPRMachineState *spapr)
 
     if (shift > 0) {
         /* Kernel handles htab, we don't need to allocate one */
+        if (shift != spapr->htab_shift) {
+            error_setg(&error_abort, "Failed to allocated requested HTAB size");
+        }
+
         spapr->htab_shift = shift;
         kvmppc_kern_htab = true;
 
Signature · 1 line
2.1.0
Reviewed-by on v0 2/2
#6 lore raw

Re: [Qemu-devel] [RFC PATCH v0 2/2] spapr: Abort when HTAB of requested size isn't allocated

Quoted 10 lines · Bharata B Rao

Terminate the guest when HTAB of requested size isn't allocated by the host.

When memory hotplug is attempted on a guest that has booted with less than requested HTAB size, the guest kernel will not be able to gracefully fail the hotplug request. This patch will ensure that we never end up in a situation where memory hotplug fails due to less than requested HTAB size.

Quoted 19 lines
---
 hw/ppc/spapr.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/ppc/spapr.c +4 -0
index 4692122..66446af 100644
@@ -992,6 +992,10 @@ static void spapr_alloc_htab(sPAPRMachineState *spapr)
 
     if (shift > 0) {
         /* Kernel handles htab, we don't need to allocate one */
+        if (shift != spapr->htab_shift) {
+            error_setg(&error_abort, "Failed to allocated requested HTAB size");
+        }
+
         spapr->htab_shift = shift;
         kvmppc_kern_htab = true;
 
Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson
#7 lore raw

Re: [Qemu-devel] [RFC PATCH v0 2/2] spapr: Abort when HTAB of requested size isn't allocated

Quoted 24 lines · David Gibson
Quoted 10 lines · Bharata B Rao

Terminate the guest when HTAB of requested size isn't allocated by the host.

When memory hotplug is attempted on a guest that has booted with less than requested HTAB size, the guest kernel will not be able to gracefully fail the hotplug request. This patch will ensure that we never end up in a situation where memory hotplug fails due to less than requested HTAB size.

Quoted 10 lines
---
 hw/ppc/spapr.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/ppc/spapr.c +0 -0
index 4692122..66446af 100644
@@ -992,6 +992,10 @@ static void spapr_alloc_htab(sPAPRMachineState *spapr)
 
Quoted 4 lines

hw/ppc/spapr.c static void spapr_alloc_htab(sPAPRMachineState *spapr) · line 996

Quoted 4 lines
hw/ppc/spapr.c +2 -0
     if (shift > 0) {
         /* Kernel handles htab, we don't need to allocate one */
+        if (shift != spapr->htab_shift) {
+            error_setg(&error_abort, "Failed to allocated requested HTAB size");

Do you think we should have a bit more verbose message here so that use knows what action to take after this abortion ? Like

"Failed to allocate requested HTAB size, try with lower maxmem"

Regards,
Bharata.

#8 lore raw

Re: [Qemu-devel] [RFC PATCH v0 2/2] spapr: Abort when HTAB of requested size isn't allocated

Quoted 33 lines · Bharata B Rao
Quoted 24 lines · David Gibson
Quoted 10 lines · Bharata B Rao

Terminate the guest when HTAB of requested size isn't allocated by the host.

When memory hotplug is attempted on a guest that has booted with less than requested HTAB size, the guest kernel will not be able to gracefully fail the hotplug request. This patch will ensure that we never end up in a situation where memory hotplug fails due to less than requested HTAB size.

Quoted 10 lines
---
 hw/ppc/spapr.c | 4 ++++
 1 file changed, 4 insertions(+)
hw/ppc/spapr.c +0 -0
index 4692122..66446af 100644
@@ -992,6 +992,10 @@ static void spapr_alloc_htab(sPAPRMachineState *spapr)
 
Quoted 4 lines

hw/ppc/spapr.c static void spapr_alloc_htab(sPAPRMachineState *spapr) · line 996

Quoted 4 lines
hw/ppc/spapr.c +2 -0
     if (shift > 0) {
         /* Kernel handles htab, we don't need to allocate one */
+        if (shift != spapr->htab_shift) {
+            error_setg(&error_abort, "Failed to allocated requested HTAB size");

Do you think we should have a bit more verbose message here so that use knows what action to take after this abortion ? Like

Quoted 1 line

"Failed to allocate requested HTAB size, try with lower maxmem"

Uh, yeah, I guess that's better.

Signature · 4 lines
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson