Rate-limiting Quality of Service

This patch enables rate-limiting quality of service (RLQOS) support as talked in the ASCAR paper [1]. The purpose of RLQOS is to provide a client-side rate limiting mechanism that controls max_rpcs_in_flight and minimal gap between brw RPC requests (called tau in the code and paper).

RLQOS can be enabled by passing --enable-rlqos to configure. It then can be controlled by tunables in procfs of each osc.

[1] http://storageconference.us/2015/Papers/14.Li.pdf

Signed-off-by Yan Li <[email protected]>

---
lustre/autoconf/lustre-core.m4 | 17 +++++++++++++++++
lustre/include/Makefile.am | 3 ++-

2 files changed, 19 insertions(+), 1 deletion(-)

lustre/autoconf/lustre-core.m4 +0 -0

index 0578325..7f1828e 100644
@@ -369,6 +369,22 @@ AC_COMPILE_IFELSE([AC_LANG_SOURCE([
 AC_MSG_RESULT([$enable_ssk])

]) # LC_OPENSSL_SSK

lustre/autoconf/lustre-core.m4 +7 -0

+#
+# LC_CONFIG_RLQOS
+#
+# Rate-limiting Quality of Service support
+#
+AC_DEFUN([LC_CONFIG_RLQOS], [
+AC_MSG_CHECKING([whether to enable rate-limiting quality of service

support])

lustre/autoconf/lustre-core.m4 +7 -0

+AC_ARG_ENABLE([rlqos],
+	AC_HELP_STRING([--enable-rlqos],
+		[enable rate-limiting quality of service support]),
+	[], [enable_rlqos="no"])
+AC_MSG_RESULT([$enable_rlqos])
+AS_IF([test "x$enable_rlqos" != xno],
+	[AC_DEFINE(ENABLE_RLQOS, 1, [enable rate-limiting quality of service

support])])
+]) # LC_CONFIG_RLQOS
+
# LC_INODE_PERMISION_2ARGS
#

# up to v2.6.27 had a 3 arg version (inode, mask, nameidata)

lustre/autoconf/lustre-core.m4 +1 -0

@@ -2241,6 +2257,7 @@ AC_DEFUN([LC_PROG_LINUX], [
 	LC_GLIBC_SUPPORT_FHANDLES
 	LC_CONFIG_GSS
 	LC_OPENSSL_SSK
+	LC_CONFIG_RLQOS
 
 	# 2.6.32
 	LC_BLK_QUEUE_MAX_SEGMENTS

lustre/include/Makefile.am +2 -1

index 9074ca4..6d72b6e 100644
@@ -98,4 +98,5 @@ EXTRA_DIST = \
 	upcall_cache.h \
 	lustre_kernelcomm.h \
 	seq_range.h \
-	uapi_kernelcomm.h
+	uapi_kernelcomm.h \
+	rlqos.h

Signature · 6 lines

1.8.3.1

_______________________________________________
lustre-devel mailing list
lustre-devel at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

I would remove the #ifdef ENABLE_RLQOS blocks, especially in lustre_idl.h since you're proposing to add new fields and consume some of the padding bits. It will cause a lot of headache for the next feature that comes along and consumes some of those bits.

Quoted 5 lines · Ben Evans

I would remove the #ifdef ENABLE_RLQOS blocks, especially in lustre_idl.h
since you're proposing to add new fields and consume some of the padding bits. It will cause a lot of headache for the next feature that comes along and consumes some of those bits.

Yeah, that's a good point. I'll remove it if all are ok with this.

Yan

This patch enables rate-limiting quality of service (RLQOS) support as talked in the ASCAR paper [1]. The purpose of RLQOS is to provide a client-side rate limiting mechanism that controls max_rpcs_in_flight and minimal gap between brw RPC requests (called tau in the code and paper).

RLQOS can be enabled by passing --enable-rlqos to configure. It then can be controlled by tunables in procfs of each osc.

[1] http://storageconference.us/2015/Papers/14.Li.pdf

Signed-off-by Yan Li <[email protected]>

---
lustre/autoconf/lustre-core.m4 | 17 +++++++++++++++++
lustre/include/Makefile.am | 3 ++-

2 files changed, 19 insertions(+), 1 deletion(-)

lustre/autoconf/lustre-core.m4 +0 -0

index 0578325..7f1828e 100644
@@ -369,6 +369,22 @@ AC_COMPILE_IFELSE([AC_LANG_SOURCE([
 AC_MSG_RESULT([$enable_ssk])

]) # LC_OPENSSL_SSK

lustre/autoconf/lustre-core.m4 +16 -0

+#
+# LC_CONFIG_RLQOS
+#
+# Rate-limiting Quality of Service support
+#
+AC_DEFUN([LC_CONFIG_RLQOS], [
+AC_MSG_CHECKING([whether to enable rate-limiting quality of service support])
+AC_ARG_ENABLE([rlqos],
+	AC_HELP_STRING([--enable-rlqos],
+		[enable rate-limiting quality of service support]),
+	[], [enable_rlqos="no"])
+AC_MSG_RESULT([$enable_rlqos])
+AS_IF([test "x$enable_rlqos" != xno],
+	[AC_DEFINE(ENABLE_RLQOS, 1, [enable rate-limiting quality of service support])])
+]) # LC_CONFIG_RLQOS
+

# LC_INODE_PERMISION_2ARGS
#

# up to v2.6.27 had a 3 arg version (inode, mask, nameidata)

lustre/autoconf/lustre-core.m4 +1 -0

@@ -2241,6 +2257,7 @@ AC_DEFUN([LC_PROG_LINUX], [
 	LC_GLIBC_SUPPORT_FHANDLES
 	LC_CONFIG_GSS
 	LC_OPENSSL_SSK
+	LC_CONFIG_RLQOS
 
 	# 2.6.32
 	LC_BLK_QUEUE_MAX_SEGMENTS

lustre/include/Makefile.am +2 -1

index 9074ca4..6d72b6e 100644
@@ -98,4 +98,5 @@ EXTRA_DIST = \
 	upcall_cache.h \
 	lustre_kernelcomm.h \
 	seq_range.h \
-	uapi_kernelcomm.h
+	uapi_kernelcomm.h \
+	rlqos.h

Signature · 6 lines

1.8.3.1

_______________________________________________
lustre-devel mailing list
lustre-devel at lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

Quoted 1 line

As you already know, we (DDN and also Prof. Andr? and Lingfang from Mainz University)

are working together on QoS, not only server side TBF policy of NRS, but also client side QoS (https://jira.hpdd.intel.com/browse/LU-7982). And also, global QoS of Lustre is under development. After a glance on the paper, I think your work looks different from our approach. That is good, because these mechanisms could work together to improve the service quality of Lustre in different ways for different requirements.

I have a few questions about LRQOS. I haven?t read all the details in the paper, so please correct me if I am wrong.

1) In my understanding, ASCAR/RLQOS is aimed at preventing congestion in the Lustre client. Am I correct? Is ASCAR/RLQOS able to provide any bandwidth/IOPS guarantees to each applications, or to each user, or job? There is an common use case of client side QoS. For example, multiple users are sharing the same Lustre client. But one of the users starts a very aggressive application which uses all the available bandwidth/IOPS and thus cause very bad performance/latency to other users. So, what we (DDN) are currently working on is to trying to isolate/balance performance between users/jobs. I am wondering whether ASCAR/RLQOS is able to be combined with our patches (https://review.whamcloud.com/#/c/19896/,
https://review.whamcloud.com/#/c/19700/) to provide an even better solution.

2) It is mentioned in the paper that ASCAR/RLQOS is controlling max flight RPC of OSC to prevent congestion. However, for cached I/O, we found that page cache limitation on client is also affecting the throughputs of applications. Especially, when multiple different applications are sharing limited page caches. One of big concern is, when max flight RPC is limited, the page cache will be exhausted (for example, when multiple applications keep on writing data), and this is a new type of congestion. I am not sure, do you think this kind of congestion will cause any performance decline/problem to the application?

3) Have you considered implementing ASCAR/RLQOS on Lustre server side? As already mentioned in the paper, sometimes, clients of Lustre could connect to the servers and send requests without any self-restraint which is unfair to other clients which follows the control of ASCAR/RLQOS. And unfortunately, it is hard to get all the clients under control since Lustre clients could change a lot from time to time. However, if a similar mechanism is implemented on server side, things becomes much easier. And that is part of the reason why TBF was implemented on server side rather than client side. Maybe something similar to ASCAR/RLQOS could be implemented on MDTs/OSTs too. What do you think?

4) In your paper, I/O patterns detection or work load classifier are mentioned. So do you know is there any any good way to detect/describe the I/O pattern of a application? Understanding the I/O patterns of applications are really important and helpful for QoS. But I guess it is really difficult, comparing to the pattern detection on other systems, like network. However, do you have any idea or direction that looks like the right way? Maybe something like machine learning?

I fully agree that your approach to learn a small rule-set is very interesting to optimize overall Lustre bandwidth. What I have not been able to fully understand from your paper is the cost of adaptation. What is happening in a cluster running many jobs at the same time applying very different access patterns (in very different combinations to different OSSes)?

We have just started to collect these patterns. Might be interesting to apply different (machine learning) algorithms on top of these patters going into different directions:

- Optimize overall bandwidth (like ASCAR is doing)

- Optimize bandwidth while supporting QoS rules for certain applications

Will you be at LUG? At least Tim from our team will participate and it might be a good opportunity to discuss a joint approach.

Modified the request message to embed sent_time, which will be returned from the server and used to calculate the exponentially weighted moving average of sent_time gap in return messages. It is used as a metric for rate-limiting quality of service.

Signed-off-by Yan Li <[email protected]>

---
 lustre/include/lustre/lustre_idl.h | 4 ++++
 lustre/ptlrpc/pack_generic.c       | 5 +++++
 lustre/ptlrpc/wiretest.c           | 2 ++
 lustre/utils/wiretest.c            | 2 ++
 4 files changed, 13 insertions(+)

lustre/include/lustre/ renamed from lustre/include/lustre/lustre_idl.h +0 -0

lustre_idl.h
index bf23a47..7a200d1 100644

lustre/include/lustre/lustre_idl.h +0 -0

@@ -3336,8 +3336,12 @@ struct obdo {
                                                 * each stripe.
                                                 * brw: grant space

consumed on
* the client for the
write */

lustre/include/lustre/lustre_idl.h +2 -0

+#ifdef ENABLE_RLQOS
+       struct timeval          o_sent_time;    /* timeval is 64x2 bits on

Linux */

lustre/include/lustre/lustre_idl.h +2 -0

+#else
        __u64                   o_padding_4;
        __u64                   o_padding_5;
+#endif
        __u64                   o_padding_6;
 };

lustre/ptlrpc/pack_generic.c +5 -0

index 8df8ea8..d0bc87a 100644
@@ -1722,8 +1722,13 @@ void lustre_swab_obdo (struct obdo  *o)
         __swab32s (&o->o_uid_h);
         __swab32s (&o->o_gid_h);
         __swab64s (&o->o_data_version);
+#ifdef ENABLE_RLQOS
+        __swab64s ((__u64*)&o->o_sent_time.tv_sec);
+        __swab64s ((__u64*)&o->o_sent_time.tv_usec);
+#else
         CLASSERT(offsetof(typeof(*o), o_padding_4) != 0);
         CLASSERT(offsetof(typeof(*o), o_padding_5) != 0);
+#endif
         CLASSERT(offsetof(typeof(*o), o_padding_6) != 0);
 
 }

lustre/ptlrpc/wiretest.c +1 -0

index 070ef91..0c909a6 100644
@@ -1314,6 +1314,7 @@ void lustre_assert_wire_constants(void)
                 (long long)(int)offsetof(struct obdo, o_data_version));
        LASSERTF((int)sizeof(((struct obdo *)0)->o_data_version) == 8,
 "found %lld\n",
                 (long long)(int)sizeof(((struct obdo
 *)0)->o_data_version));
+#ifndef ENABLE_RLQOS
        LASSERTF((int)offsetof(struct obdo, o_padding_4) == 184, "found

%lld\n",

                 (long long)(int)offsetof(struct obdo, o_padding_4));
        LASSERTF((int)sizeof(((struct obdo *)0)->o_padding_4) == 8, "found

%lld\n",

lustre/ptlrpc/wiretest.c +1 -0

@@ -1322,6 +1323,7 @@ void lustre_assert_wire_constants(void)
                 (long long)(int)offsetof(struct obdo, o_padding_5));
        LASSERTF((int)sizeof(((struct obdo *)0)->o_padding_5) == 8, "found
 %lld\n",
                 (long long)(int)sizeof(((struct obdo *)0)->o_padding_5));
+#endif
        LASSERTF((int)offsetof(struct obdo, o_padding_6) == 200, "found
 %lld\n",

                 (long long)(int)offsetof(struct obdo, o_padding_6));
        LASSERTF((int)sizeof(((struct obdo *)0)->o_padding_6) == 8, "found

%lld\n",

lustre/utils/wiretest.c +1 -0

index 233d7d8..47fbbf0 100644
@@ -1329,6 +1329,7 @@ void lustre_assert_wire_constants(void)
                 (long long)(int)offsetof(struct obdo, o_data_version));
        LASSERTF((int)sizeof(((struct obdo *)0)->o_data_version) == 8,
 "found %lld\n",
                 (long long)(int)sizeof(((struct obdo
 *)0)->o_data_version));
+#ifndef ENABLE_RLQOS
        LASSERTF((int)offsetof(struct obdo, o_padding_4) == 184, "found

%lld\n",

                 (long long)(int)offsetof(struct obdo, o_padding_4));
        LASSERTF((int)sizeof(((struct obdo *)0)->o_padding_4) == 8, "found

%lld\n",

lustre/utils/wiretest.c +1 -0

@@ -1337,6 +1338,7 @@ void lustre_assert_wire_constants(void)
                 (long long)(int)offsetof(struct obdo, o_padding_5));
        LASSERTF((int)sizeof(((struct obdo *)0)->o_padding_5) == 8, "found
 %lld\n",
                 (long long)(int)sizeof(((struct obdo *)0)->o_padding_5));
+#endif
        LASSERTF((int)offsetof(struct obdo, o_padding_6) == 200, "found
 %lld\n",

                 (long long)(int)offsetof(struct obdo, o_padding_6));
        LASSERTF((int)sizeof(((struct obdo *)0)->o_padding_6) == 8, "found

%lld\n",
--
1.8.3.1

_______________________________________________
lustre-devel mailing list
lustre-devel at lists.lustre.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.
lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=DwICAg&c=IGDlg0lD0b- nebmJJ0Kp8A&r=m8P9AM2wTf4l79yg9e1LHD5IHagtwa3P4AXaemlM6Lg&m= NuClc8LkPaQ91Zav0h5yoiRmBVC4_Ks9Db6KX3xsRmk&s= 6FVNfemWTMvnOwmVBxixoJyS4CNIP_D14UGw2pWlGd0&e=

Signed-off-by Yan Li <[email protected]>

---
 lustre/osc/osc_cache.c    |  3 +++
 lustre/osc/osc_internal.h | 66 ++++++++++++++++++++++++++++++
+++++++++++++++++
 2 files changed, 69 insertions(+)

lustre/osc/osc_cache.c +3 -0

index 236263c..2f9d4e1 100644
@@ -2316,6 +2316,9 @@ static int osc_io_unplug0(const struct lu_env *env,
 struct client_obd *cli,
        } else {
                CDEBUG(D_CACHE, "Queue writeback work for client %p.\n",
 cli);
                LASSERT(cli->cl_writeback_work != NULL);
+#ifdef ENABLE_RLQOS
+               qos_throttle(&cli->qos);
+#endif
                rc = ptlrpcd_queue_work(cli->cl_writeback_work);

        }
        return rc;

lustre/osc/osc_internal.h +13 -0

index 06c21b3..d31d5ba 100644
@@ -245,4 +245,70 @@ extern unsigned long osc_cache_shrink_count(struct
 shrinker *sk,
 extern unsigned long osc_cache_shrink_scan(struct shrinker *sk,
                                           struct shrink_control *sc);
 
+#ifdef ENABLE_RLQOS
+static inline void qos_throttle(struct qos_data_t *qos)
+{
+       struct timeval now;
+       long           usec_since_last_rpc;
+       long           need_sleep_usec = 0;
+
+       spin_lock(&qos->lock);
+       if (0 == qos->min_usec_between_rpcs)
+               goto out;
+
+       do_gettimeofday(&now);
+       usec_since_last_rpc = cfs_timeval_sub(&now, &qos->last_rpc_time,

NULL);

lustre/osc/osc_internal.h +5 -0

+       if (usec_since_last_rpc < 0) {
+               usec_since_last_rpc = 0;
+       }
+       if (usec_since_last_rpc < qos->min_usec_between_rpcs) {
+               need_sleep_usec = qos->min_usec_between_rpcs -

usec_since_last_rpc;

lustre/osc/osc_internal.h +10 -0

+       }
+       qos->last_rpc_time = now;
+out:
+       spin_unlock(&qos->lock);
+       if (0 == need_sleep_usec) {
+               return;
+       }
+
+       /* About timer ranges:
+          Ref: https://urldefense.proofpoint.com/v2/url?u=https-3A__www.

kernel.org_doc_Documentation_timers_timers-2Dhowto.txt&d= DwICAg&c=IGDlg0lD0b-nebmJJ0Kp8A&r=m8P9AM2wTf4l79yg9e1LHD5IHagtwa 3P4AXaemlM6Lg&m=w0oijGmz2ea38--CHGZq4fPu44dwEldJr2BDVZcBR2U& s=jN5WjVQ8jELL9iEXADWoal4-Yo76FIU3VVDcdN3zsC4&e= */

lustre/osc/osc_internal.h +12 -0

+       if (need_sleep_usec < 1000) {
+               udelay(need_sleep_usec);
+       } else if (need_sleep_usec < 20000) {
+               usleep_range(need_sleep_usec - 1, need_sleep_usec);
+       } else {
+               msleep(need_sleep_usec / 1000);
+       }
+}
+#endif /* ENABLE_RLQOS */
+
+/* You must call LPROCFS_CLIMP_CHECK() on the obd device before and
+ * LPROCFS_CLIMP_EXIT() after calling this function. They are not called

inside
+ * this function, because they may return an error code.
+ */

lustre/osc/osc_internal.h +24 -1

+static inline void set_max_rpcs_in_flight(int val, struct client_obd *cli)
+{
+       int adding, added, req_count;
+
+       adding = val - cli->cl_max_rpcs_in_flight;
+       req_count = atomic_read(&osc_pool_req_count);
+       if (adding > 0 && req_count < osc_reqpool_maxreqcount) {
+               /*
+                * There might be some race which will cause over-limit
+                * allocation, but it is fine.
+                */
+               if (req_count + adding > osc_reqpool_maxreqcount)
+                       adding = osc_reqpool_maxreqcount - req_count;
+
+               added = osc_rq_pool->prp_populate(osc_rq_pool, adding);
+               atomic_add(added, &osc_pool_req_count);
+       }
+
+       spin_lock(&cli->cl_loi_list_lock);
+       cli->cl_max_rpcs_in_flight = val;
+       client_adjust_max_dirty(cli);
+       spin_unlock(&cli->cl_loi_list_lock);
+}
+
 #endif /* OSC_INTERNAL_H */
--

1.8.3.1

_______________________________________________
lustre-devel mailing list
lustre-devel at lists.lustre.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.
lustre.org_listinfo.cgi_lustre-2Ddevel-2Dlustre.org&d=DwICAg&c=IGDlg0lD0b- nebmJJ0Kp8A&r=m8P9AM2wTf4l79yg9e1LHD5IHagtwa3P4AXaemlM6Lg&m=w0oijGmz2ea38- -CHGZq4fPu44dwEldJr2BDVZcBR2U&s=ppAA2u9phKTaqwpnFsNVQGtqbG3xF6 tk4_Q9mVL_lGk&e=