From: Dan Magenheimer Subject: [PATCH] tmem: reclaim minimal memory proactively Date: Tue, 8 Dec 2009 11:16:38 -0800 (PST) Message-ID: <687d3994-3764-4748-b31e-eff287c3bcc9@default> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="__126029983719615738abhmt014.oracle.com" Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: "Xen-Devel (E-mail)" List-Id: xen-devel@lists.xenproject.org --__126029983719615738abhmt014.oracle.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable When a single domain is using most/all of tmem memory for ephemeral pages belonging to the same object, e.g. when copying a single huge file larger than ephemeral memory, long lists are traversed looking for a page to evict that doesn't belong to this object (as pages in the object for which a page is currently being inserted are locked and cannot be evicted). This is essentially a livelock. Avoid this by proactively ensuring there is a margin of available memory (1MB) before locks are taken on the object. Signed-off-by: Dan Magenheimer diff -r 7f611de6b93c xen/common/tmem.c --- a/xen/common/tmem.c=09Tue Dec 08 14:14:27 2009 +0000 +++ b/xen/common/tmem.c=09Tue Dec 08 12:09:05 2009 -0700 @@ -1091,6 +1091,24 @@ static unsigned long tmem_relinquish_npa if ( avail_pages ) tmh_release_avail_pages_to_host(); return avail_pages; +} + +/* Under certain conditions (e.g. if each client is putting pages for exac= tly + * one object), once locks are held, freeing up memory may + * result in livelocks and very long "put" times, so we try to ensure ther= e + * is a minimum amount of memory (1MB) available BEFORE any data structure + * locks are held */ +static inline void tmem_ensure_avail_pages(void) +{ + int failed_evict =3D 10; + + while ( !tmh_free_mb() ) + { + if ( tmem_evict() ) + continue; + else if ( failed_evict-- <=3D 0 ) + break; + } } =20 /************ TMEM CORE OPERATIONS ************************************/ @@ -2315,10 +2333,12 @@ EXPORT long do_tmem_op(tmem_cli_op_t uop op.u.new.uuid[0], op.u.new.uuid[1]); break; case TMEM_NEW_PAGE: + tmem_ensure_avail_pages(); rc =3D do_tmem_put(pool, op.u.gen.object, op.u.gen.index, op.u.gen.cmfn, 0, 0, 0, NULL); break; case TMEM_PUT_PAGE: + tmem_ensure_avail_pages(); rc =3D do_tmem_put(pool, op.u.gen.object, op.u.gen.index, op.u.gen.cmfn, 0, 0, PAGE_SIZE, NULL); if (rc =3D=3D 1) succ_put =3D 1; diff -r 7f611de6b93c xen/include/xen/tmem_xen.h --- a/xen/include/xen/tmem_xen.h=09Tue Dec 08 14:14:27 2009 +0000 +++ b/xen/include/xen/tmem_xen.h=09Tue Dec 08 12:09:05 2009 -0700 @@ -252,6 +252,11 @@ static inline unsigned long tmh_freeable (20 - PAGE_SHIFT); } =20 +static inline unsigned long tmh_free_mb(void) +{ + return (tmh_avail_pages() + total_free_pages()) >> (20 - PAGE_SHIFT); +} + /* * Memory allocation for "infrastructure" data */ --__126029983719615738abhmt014.oracle.com Content-Type: application/octet-stream; name="tmem-livelock.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="tmem-livelock.patch" ZGlmZiAtciA3ZjYxMWRlNmI5M2MgeGVuL2NvbW1vbi90bWVtLmMKLS0tIGEveGVuL2NvbW1vbi90 bWVtLmMJVHVlIERlYyAwOCAxNDoxNDoyNyAyMDA5ICswMDAwCisrKyBiL3hlbi9jb21tb24vdG1l bS5jCVR1ZSBEZWMgMDggMTI6MDk6MDUgMjAwOSAtMDcwMApAQCAtMTA5MSw2ICsxMDkxLDI0IEBA IHN0YXRpYyB1bnNpZ25lZCBsb25nIHRtZW1fcmVsaW5xdWlzaF9ucGEKICAgICBpZiAoIGF2YWls X3BhZ2VzICkKICAgICAgICAgdG1oX3JlbGVhc2VfYXZhaWxfcGFnZXNfdG9faG9zdCgpOwogICAg IHJldHVybiBhdmFpbF9wYWdlczsKK30KKworLyogVW5kZXIgY2VydGFpbiBjb25kaXRpb25zIChl LmcuIGlmIGVhY2ggY2xpZW50IGlzIHB1dHRpbmcgcGFnZXMgZm9yIGV4YWN0bHkKKyAqIG9uZSBv YmplY3QpLCBvbmNlIGxvY2tzIGFyZSBoZWxkLCBmcmVlaW5nIHVwIG1lbW9yeSBtYXkKKyAqIHJl c3VsdCBpbiBsaXZlbG9ja3MgYW5kIHZlcnkgbG9uZyAicHV0IiB0aW1lcywgc28gd2UgdHJ5IHRv IGVuc3VyZSB0aGVyZQorICogaXMgYSBtaW5pbXVtIGFtb3VudCBvZiBtZW1vcnkgKDFNQikgYXZh aWxhYmxlIEJFRk9SRSBhbnkgZGF0YSBzdHJ1Y3R1cmUKKyAqIGxvY2tzIGFyZSBoZWxkICovCitz dGF0aWMgaW5saW5lIHZvaWQgdG1lbV9lbnN1cmVfYXZhaWxfcGFnZXModm9pZCkKK3sKKyAgICBp bnQgZmFpbGVkX2V2aWN0ID0gMTA7CisKKyAgICB3aGlsZSAoICF0bWhfZnJlZV9tYigpICkKKyAg ICB7CisgICAgICAgIGlmICggdG1lbV9ldmljdCgpICkKKyAgICAgICAgICAgIGNvbnRpbnVlOwor ICAgICAgICBlbHNlIGlmICggZmFpbGVkX2V2aWN0LS0gPD0gMCApCisgICAgICAgICAgICBicmVh azsKKyAgICB9CiB9CiAKIC8qKioqKioqKioqKiogVE1FTSBDT1JFIE9QRVJBVElPTlMgKioqKioq KioqKioqKioqKioqKioqKioqKioqKioqKioqKioqLwpAQCAtMjMxNSwxMCArMjMzMywxMiBAQCBF WFBPUlQgbG9uZyBkb190bWVtX29wKHRtZW1fY2xpX29wX3QgdW9wCiAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICBvcC51Lm5ldy51dWlkWzBdLCBvcC51Lm5ldy51dWlkWzFdKTsKICAgICAg ICAgYnJlYWs7CiAgICAgY2FzZSBUTUVNX05FV19QQUdFOgorICAgICAgICB0bWVtX2Vuc3VyZV9h dmFpbF9wYWdlcygpOwogICAgICAgICByYyA9IGRvX3RtZW1fcHV0KHBvb2wsIG9wLnUuZ2VuLm9i amVjdCwKICAgICAgICAgICAgICAgICAgICAgICAgICBvcC51Lmdlbi5pbmRleCwgb3AudS5nZW4u Y21mbiwgMCwgMCwgMCwgTlVMTCk7CiAgICAgICAgIGJyZWFrOwogICAgIGNhc2UgVE1FTV9QVVRf UEFHRToKKyAgICAgICAgdG1lbV9lbnN1cmVfYXZhaWxfcGFnZXMoKTsKICAgICAgICAgcmMgPSBk b190bWVtX3B1dChwb29sLCBvcC51Lmdlbi5vYmplY3QsCiAgICAgICAgICAgICAgICAgICAgIG9w LnUuZ2VuLmluZGV4LCBvcC51Lmdlbi5jbWZuLCAwLCAwLCBQQUdFX1NJWkUsIE5VTEwpOwogICAg ICAgICBpZiAocmMgPT0gMSkgc3VjY19wdXQgPSAxOwpkaWZmIC1yIDdmNjExZGU2YjkzYyB4ZW4v aW5jbHVkZS94ZW4vdG1lbV94ZW4uaAotLS0gYS94ZW4vaW5jbHVkZS94ZW4vdG1lbV94ZW4uaAlU dWUgRGVjIDA4IDE0OjE0OjI3IDIwMDkgKzAwMDAKKysrIGIveGVuL2luY2x1ZGUveGVuL3RtZW1f eGVuLmgJVHVlIERlYyAwOCAxMjowOTowNSAyMDA5IC0wNzAwCkBAIC0yNTIsNiArMjUyLDExIEBA IHN0YXRpYyBpbmxpbmUgdW5zaWduZWQgbG9uZyB0bWhfZnJlZWFibGUKICAgICAgICAgICAgICgy MCAtIFBBR0VfU0hJRlQpOwogfQogCitzdGF0aWMgaW5saW5lIHVuc2lnbmVkIGxvbmcgdG1oX2Zy ZWVfbWIodm9pZCkKK3sKKyAgICByZXR1cm4gKHRtaF9hdmFpbF9wYWdlcygpICsgdG90YWxfZnJl ZV9wYWdlcygpKSA+PiAoMjAgLSBQQUdFX1NISUZUKTsKK30KKwogLyoKICAqIE1lbW9yeSBhbGxv Y2F0aW9uIGZvciAiaW5mcmFzdHJ1Y3R1cmUiIGRhdGEKICAqLwo= --__126029983719615738abhmt014.oracle.com Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --__126029983719615738abhmt014.oracle.com--