Skip to contentSkip to conversation

x86/domctl: Have XEN_DOMCTL_getpageframeinfo3 preemptible

PATCH Superseded Anthony PERARD 2 participants 2 messages Nov 19, 2019

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/domctl: Have XEN_DOMCTL_getpageframeinfo3 preemptible

This hypercall can take a long time to finish because it attempts to grab the `hostp2m' lock up to 1024 times. The accumulated wait for the lock can take several seconds.

This can easily happen with a guest with 32 vcpus and plenty of RAM, during localhost migration.

---

Notes:
I don't know if it's a correct way to make the hypercall preemptible, the patch kind of modify the response, but libxc doesn't seems to care.

Is it fine to modify the domctl_t that the domain (dom0) provides? If not, where could we store the progress made?

 xen/arch/x86/domctl.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
xen/arch/x86/domctl.c +12 -0
index 43e368d63bb9..5c0a7462e63b 100644
@@ -425,6 +425,18 @@ long arch_do_domctl(
                 ret = -EFAULT;
                 break;
             }
+
+            if ( hypercall_preempt_check() ) {
+                domctl->u.getpageframeinfo3.num = num - i - 1;
+                domctl->u.getpageframeinfo3.array.p =
+                    guest_handle + ((i + 1) * width);
+                if ( __copy_to_guest(u_domctl, domctl, 1) ) {
+                    ret = -EFAULT;
+                    break;
+                }
+                return hypercall_create_continuation(__HYPERVISOR_domctl,
+                                                     "h", u_domctl);
+            }
         }
 
         break;
Signature · 7 lines
Anthony PERARD


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw
Quoted 3 lines · Anthony PERARD

This hypercall can take a long time to finish because it attempts to grab the `hostp2m' lock up to 1024 times. The accumulated wait for the lock can take several seconds.

Which means several milliseconds on average per lock acquire. This points (again) at a (the) bigger problem of p2m lock contention. Therefore I'm afraid that, while the change here is an improvement, it's only curing symptoms rather than the cause.

Seeing that p2m_get_page_from_gfn() uses a read lock (but sadly is the only function doing so), one route could be to investigate whether further paths could get away with just read locking. Fair parts of e.g. the nested page fault handling don't really seem to require a write lock, but there is this comment

/*
* Take a lock on the host p2m speculatively, to avoid potential
* locking order problems later and to handle unshare etc. */

pointing out the possible issues with downgrading the lock to just a read one.

Another route to consider would be to avoid taking the lock once per iteration, and instead process several GFNs at a time within a single lock holding sequence.

Quoted 3 lines

Notes:
I don't know if it's a correct way to make the hypercall preemptible, the patch kind of modify the response, but libxc doesn't seems to care.

I think that's acceptable for domctl-s, but would better be accompanied by a comment adjustment/addition to public/domctl.h.

xen/arch/x86/domctl.c long arch_do_domctl( · line 429

Quoted 8 lines
xen/arch/x86/domctl.c +2 -0
@@ -425,6 +425,18 @@ long arch_do_domctl(
                 ret = -EFAULT;
                 break;
             }
+
+            if ( hypercall_preempt_check() ) {

You don't want this on the last iteration. You also better don't do this when there's no p2m lock involved to begin with, i.e. for non-translated guests. This should then be accompanied by a comment justifying the special casing.

Also the opening brace (one more below here) goes on its own line.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel