detect and initialize CDP (Code/Data Prioritization) feature

PATCH Superseded He Chen 3 participants 8 messages Sep 28, 2015 – Sep 30, 2015

Share & tools

He Chen

author

2015-09-28 08:29 UTC #0 lore raw

v5 cover detect and initialize CDP (Code/Data Prioritization) feature

Changes in v5:
- x86: address Andrew's and Jan's coments.
- tools: refine options parsing in psr-cat-cbm-set and invalidate passing -c and -d at the same time
- merge tools and docs patches.
- code style

Changes in v4:
- x86:
* remove union member name in struct psr_cat_cbm (suggested by Jan)
* fix log info of CAT & CDP (suggested by Chao & Jan)
* add a helper cdp_is_enabled to tell the status of CDP and CDP initialize failed is considered (Jan's comment)
* XEN_SYSCTL_INTERFACE_VERSION 0x0000000C -> 0x0000000D (suggested by Jan)
* refine CBM type check logic in get/set CBM function (suggested by Jan)
* loop optimization in function find_cos (suggested by Jan)
- tools: address Chao's comments.
- docs: address Chao's comments.
- code style

Changes in v3:
- x86: remove redundant CDP field in cat_socket_enable (suggested by Chao)
- tools: simplify CBM setting function in tools (suggested by Jan)
- docs: add boot parameter description (suggested by Chao & Ian)
- code style

Changes in v2:
- x86: enable CDP by boot parameter instead of enabling/disabling CDP at runtime (suggested by Andrew)
- tools: remove psr-cat-cdp-enable/disable xl commands
- code style

Code/Data Prioritization(CDP) is offered in Intel Broadwell and later server platforms, which is an extension of CAT. CDP enables isolation and separate prioritization of code and data fetches to the L3 cache in a software configurable manner, which can enable workload prioritization and tuning of cache capacity to the characteristics of the workload. CDP extends Cache Allocation Technology (CAT) by providing separate code and data capacity bit masks(CBM) per Class of Service (COS). CDP is used on VM basis in the Xen implementation.

More information about CDP, please refer to Intel SDM, Volumn 3, section 17.16
http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-manual-325462.pdf

This patch series enables CDP feature in Xen based on CAT code, and extends CBM operation functions to support CDP. For all the changes, please see in each patch.

This v5 patchset has been tested on Intel Broadwell server platform.

To make this patchset better, any comment or suggestion is welcomed, I would really appreciate it.

Thanks.

He Chen (3):
x86: Support enable CDP by boot parameter and add get CDP status
x86: add domctl cmd to set/get CDP code/data CBM tools & docs: add tools and docs support for Intel CDP

 docs/man/xl.pod.1                   |  15 +++
 docs/misc/xen-command-line.markdown |  11 +-
 docs/misc/xl-psr.markdown           |  47 ++++++-
 tools/libxc/include/xenctrl.h       |   7 +-
 tools/libxc/xc_psr.c                |  17 ++-
 tools/libxl/libxl.h                 |   7 ++
 tools/libxl/libxl_psr.c             |   5 +-
 tools/libxl/libxl_types.idl         |   3 +
 tools/libxl/xl_cmdimpl.c            |  56 +++++++--
 tools/libxl/xl_cmdtable.c           |   2 +
 xen/arch/x86/domctl.c               |  32 ++++-
 xen/arch/x86/psr.c                  | 237 +++++++++++++++++++++++++++++-------
 xen/arch/x86/sysctl.c               |   5 +-
 xen/include/asm-x86/msr-index.h     |   3 +
 xen/include/asm-x86/psr.h           |  20 ++-
 xen/include/public/domctl.h         |   4 +
 xen/include/public/sysctl.h         |   4 +-
 17 files changed, 398 insertions(+), 77 deletions(-)

Signature · 1 line

1.9.1

He Chen

author

2015-09-28 08:29 UTC #1 lore raw

PATCH v5 1/3 x86: Support enable CDP by boot parameter and add get CDP status

Add boot parameter psr=cdp to enable CDP at boot time. Intel Code/Data Prioritization (CDP) feature is based on CAT. Note that cos_max would be half when CDP is on. struct psr_cat_cbm is extended to support CDP operation. Extend psr_get_cat_l3_info sysctl to get CDP status.

Signed-off-by He Chen <[email protected]>

Reviewed-by Andrew Cooper <[email protected]>

---
Changes in v5
* remove unnecessary u in psr_cat_cbm structure
* revert write_l3_cbm and put the modification to next patch
* remove duplicate PSR_CAT_FLAG_L3_CDP

---
 docs/misc/xen-command-line.markdown | 11 +++++++--
 xen/arch/x86/psr.c                  | 49 ++++++++++++++++++++++++++++++++++---
 xen/arch/x86/sysctl.c               |  5 ++--
 xen/include/asm-x86/msr-index.h     |  3 +++
 xen/include/asm-x86/psr.h           |  8 +++++-
 xen/include/public/sysctl.h         |  4 ++-
 6 files changed, 70 insertions(+), 10 deletions(-)

docs/misc/xen-command-line.markdown +9 -2

index a2e427c..d92e323 100644
@@ -1165,9 +1165,9 @@ This option can be specified more than once (up to 8 times at present).
 > `= <integer>`
 
 ### psr (Intel)
-> `= List of ( cmt:<boolean> | rmid_max:<integer> | cat:<boolean> | cos_max:<integer> )`
+> `= List of ( cmt:<boolean> | rmid_max:<integer> | cat:<boolean> | cos_max:<integer> | cdp:<boolean> )`
 
-> Default: `psr=cmt:0,rmid_max:255,cat:0,cos_max:255`
+> Default: `psr=cmt:0,rmid_max:255,cat:0,cos_max:255,cdp:0`
 
 Platform Shared Resource(PSR) Services.  Intel Haswell and later server
 platforms offer information about the sharing of resources.

@@ -1197,6 +1197,13 @@ The following resources are available:
   the cache allocation.
   * `cat` instructs Xen to enable/disable Cache Allocation Technology.
   * `cos_max` indicates the max value for COS ID.
+* Code and Data Prioritization Technology (Broadwell and later). Information
+  regarding the code cache and the data cache allocation. CDP is based on CAT.
+  * `cdp` instructs Xen to enable/disable Code and Data Prioritization. Note
+    that `cos_max` of CDP is a little different from `cos_max` of CAT. With
+    CDP, one COS will corespond two CBMs other than one with CAT, due to the
+    sum of CBMs is fixed, that means actual `cos_max` in use will automatically
+    reduce to half when CDP is enabled.
 
 ### reboot
 > `= t[riple] | k[bd] | a[cpi] | p[ci] | P[ower] | e[fi] | n[o] [, [w]arm | [c]old]`

xen/arch/x86/psr.c +45 -4

106 lines not on this page · raw message

xen/arch/x86/sysctl.c +3 -2

index f36b52f..e42c5e7 100644
@@ -177,12 +177,13 @@ long arch_do_sysctl(
         case XEN_SYSCTL_PSR_CAT_get_l3_info:
             ret = psr_get_cat_l3_info(sysctl->u.psr_cat_op.target,
                                       &sysctl->u.psr_cat_op.u.l3_info.cbm_len,
-                                      &sysctl->u.psr_cat_op.u.l3_info.cos_max);
+                                      &sysctl->u.psr_cat_op.u.l3_info.cos_max,
+                                      &sysctl->u.psr_cat_op.u.l3_info.flags);
 
             if ( !ret && __copy_field_to_guest(u_sysctl, sysctl, u.psr_cat_op) )
                 ret = -EFAULT;
-
             break;
+
         default:
             ret = -EOPNOTSUPP;
             break;

xen/include/asm-x86/msr-index.h +3 -0

index e9c4723..65c1d02 100644
@@ -328,7 +328,10 @@
 #define MSR_IA32_CMT_EVTSEL		0x00000c8d
 #define MSR_IA32_CMT_CTR		0x00000c8e
 #define MSR_IA32_PSR_ASSOC		0x00000c8f
+#define MSR_IA32_PSR_L3_QOS_CFG	0x00000c81
 #define MSR_IA32_PSR_L3_MASK(n)	(0x00000c90 + (n))
+#define MSR_IA32_PSR_L3_MASK_CODE(n)	(0x00000c90 + (n) * 2 + 1)
+#define MSR_IA32_PSR_L3_MASK_DATA(n)	(0x00000c90 + (n) * 2)
 
 /* Intel Model 6 */
 #define MSR_P6_PERFCTR(n)		(0x000000c1 + (n))

xen/include/asm-x86/psr.h +7 -1

index 081750f..5faeaaf 100644
@@ -27,6 +27,12 @@
 /* L3 Monitoring Features */
 #define PSR_CMT_L3_OCCUPANCY           0x1
 
+/* CDP Capability */
+#define PSR_CAT_CDP_CAPABILITY       (1u << 2)
+
+/* L3 CDP Enable bit*/
+#define PSR_L3_QOS_CDP_ENABLE_BIT       0x0
+
 struct psr_cmt_l3 {
     unsigned int features;
     unsigned int upscaling_factor;

@@ -52,7 +58,7 @@ void psr_free_rmid(struct domain *d);
 void psr_ctxt_switch_to(struct domain *d);
 
 int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
-                        uint32_t *cos_max);
+                        uint32_t *cos_max, uint32_t *flags);
 int psr_get_l3_cbm(struct domain *d, unsigned int socket, uint64_t *cbm);
 int psr_set_l3_cbm(struct domain *d, unsigned int socket, uint64_t cbm);

xen/include/public/sysctl.h +3 -1

index 0cacacc..96680eb 100644
@@ -36,7 +36,7 @@
 #include "physdev.h"
 #include "tmem.h"
 
-#define XEN_SYSCTL_INTERFACE_VERSION 0x0000000C
+#define XEN_SYSCTL_INTERFACE_VERSION 0x0000000D
 
 /*
  * Read console content from Xen buffer ring.

@@ -705,6 +705,8 @@ struct xen_sysctl_psr_cat_op {
         struct {
             uint32_t cbm_len;   /* OUT: CBM length */
             uint32_t cos_max;   /* OUT: Maximum COS */
+#define XEN_SYSCTL_PSR_CAT_L3_CDP       (1u << 0)
+            uint32_t flags;     /* OUT: CAT flags */
         } l3_info;
     } u;
 };

Signature · 1 line

1.9.1

Chao Peng

2015-09-30 03:14 UTC #2 lore raw

Re: [PATCH v5 1/3] x86: Support enable CDP by boot parameter and add get CDP status

xen/arch/x86/psr.c static struct psr_cat_socket_info *get_cat_socket_info(unsigned int socket) · line 274

Quoted 3 lines

xen/arch/x86/psr.c +2 -0

+static inline bool_t cdp_is_enabled(unsigned int socket,
+                                    unsigned long *cdp_socket_enable)

Since cdp_socket_enable is defined in file scope, I don't think it's required to pass again.

Chao

Quoted 3 lines

xen/arch/x86/psr.c +3 -0

+{
+    return cdp_socket_enable && test_bit(socket, cdp_socket_enable);
+}

He Chen

author

2015-09-28 08:29 UTC #3 lore raw

PATCH v5 2/3 x86: add domctl cmd to set/get CDP code/data CBM

In reply to He Chen

CDP extends CAT and provides the capacity to control L3 code & data cache. With CDP, one COS corresponds to two CMBs(code & data). cbm_type is added to distinguish different CBM operations. Besides, new domctl cmds are introdunced to support set/get CDP CBM. Some CAT functions to operation CBMs are extended to support CDP.

Signed-off-by He Chen <[email protected]>

Reviewed-by Andrew Cooper <[email protected]>

---
Changes in v5:
* replace -EINVAL with -ENXIO when setting code/data CBM on CDP disabled.

---
 xen/arch/x86/domctl.c       |  32 +++++++-
 xen/arch/x86/psr.c          | 188 ++++++++++++++++++++++++++++++++++----------
 xen/include/asm-x86/psr.h   |  12 ++-
 xen/include/public/domctl.h |   4 +
 4 files changed, 191 insertions(+), 45 deletions(-)

xen/arch/x86/domctl.c +30 -2

index bf62a88..734fddb 100644
@@ -1167,12 +1167,40 @@ long arch_do_domctl(
         {
         case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM:
             ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
-                                 domctl->u.psr_cat_op.data);
+                                 domctl->u.psr_cat_op.data,
+                                 PSR_CBM_TYPE_L3);
+            break;
+
+        case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE:
+            ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
+                                 domctl->u.psr_cat_op.data,
+                                 PSR_CBM_TYPE_L3_CODE);
+            break;
+
+        case XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA:
+            ret = psr_set_l3_cbm(d, domctl->u.psr_cat_op.target,
+                                 domctl->u.psr_cat_op.data,
+                                 PSR_CBM_TYPE_L3_DATA);
             break;
 
         case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM:
             ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
-                                 &domctl->u.psr_cat_op.data);
+                                 &domctl->u.psr_cat_op.data,
+                                 PSR_CBM_TYPE_L3);
+            copyback = 1;
+            break;
+
+        case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE:
+            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
+                                 &domctl->u.psr_cat_op.data,
+                                 PSR_CBM_TYPE_L3_CODE);
+            copyback = 1;
+            break;
+
+        case XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA:
+            ret = psr_get_l3_cbm(d, domctl->u.psr_cat_op.target,
+                                 &domctl->u.psr_cat_op.data,
+                                 PSR_CBM_TYPE_L3_DATA);
             copyback = 1;
             break;

xen/arch/x86/psr.c +147 ~~-41~~

243 lines not on this page · raw message

xen/include/asm-x86/psr.h +10 -2

index 5faeaaf..57f47e9 100644
@@ -46,6 +46,12 @@ struct psr_cmt {
     struct psr_cmt_l3 l3;
 };
 
+enum cbm_type {
+    PSR_CBM_TYPE_L3,
+    PSR_CBM_TYPE_L3_CODE,
+    PSR_CBM_TYPE_L3_DATA,
+};
+
 extern struct psr_cmt *psr_cmt;
 
 static inline bool_t psr_cmt_enabled(void)

@@ -59,8 +65,10 @@ void psr_ctxt_switch_to(struct domain *d);
 
 int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len,
                         uint32_t *cos_max, uint32_t *flags);
-int psr_get_l3_cbm(struct domain *d, unsigned int socket, uint64_t *cbm);
-int psr_set_l3_cbm(struct domain *d, unsigned int socket, uint64_t cbm);
+int psr_get_l3_cbm(struct domain *d, unsigned int socket,
+                   uint64_t *cbm, enum cbm_type type);
+int psr_set_l3_cbm(struct domain *d, unsigned int socket,
+                   uint64_t cbm, enum cbm_type type);
 
 int psr_domain_init(struct domain *d);
 void psr_domain_free(struct domain *d);

xen/include/public/domctl.h +4 -0

index 675f021..f438cae 100644
@@ -1056,6 +1056,10 @@ DEFINE_XEN_GUEST_HANDLE(xen_domctl_monitor_op_t);
 struct xen_domctl_psr_cat_op {
 #define XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM     0
 #define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM     1
+#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE    2
+#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA    3
+#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE    4
+#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA    5
     uint32_t cmd;       /* IN: XEN_DOMCTL_PSR_CAT_OP_* */
     uint32_t target;    /* IN */
     uint64_t data;      /* IN/OUT */

Signature · 1 line

1.9.1

Chao Peng

2015-09-30 03:34 UTC #4 lore raw

Re: [PATCH v5 2/3] x86: add domctl cmd to set/get CDP code/data CBM

xen/arch/x86/psr.c int psr_get_cat_l3_info(unsigned int socket, uint32_t *cbm_len, · line 309

Quoted 10 lines

xen/arch/x86/psr.c +5 -1

+    bool_t cdp_enabled = cdp_is_enabled(socket, cdp_socket_enable);
 
     if ( IS_ERR(info) )
         return PTR_ERR(info);
 
-    *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
+    switch ( type )
+    {
+    case PSR_CBM_TYPE_L3:
+        if ( type == PSR_CBM_TYPE_L3 && cdp_enabled )

'type == PSR_CBM_TYPE_L3' here seems redundant.

Quoted 3 lines

xen/arch/x86/psr.c +3 -0

+            return -EXDEV;
+        *cbm = info->cos_to_cbm[d->arch.psr_cos_ids[socket]].cbm;
+        break;

...

xen/arch/x86/psr.c static int write_l3_cbm(unsigned int socket, unsigned int cos, uint64_t cbm) · line 404

Quoted 2 lines

xen/arch/x86/psr.c +1 -1

-int psr_set_l3_cbm(struct domain *d, unsigned int socket, uint64_t cbm)
+static int find_cos(struct psr_cat_cbm *map, int cos_max,

cos_max should be unsigned.

Quoted 15 lines

xen/arch/x86/psr.c +12 -2

+                    uint64_t cbm_code, uint64_t cbm_data, bool_t cdp_enabled)
 {
-    unsigned int old_cos, cos;
-    struct psr_cat_cbm *map, *found = NULL;
+    unsigned int cos;
+
+    for ( cos = 0; cos <= cos_max; cos++ )
+    {
+        if( map[cos].ref &&
+            ((!cdp_enabled && map[cos].cbm == cbm_code) ||
+             (cdp_enabled && map[cos].code == cbm_code &&
+                             map[cos].data == cbm_data)))
+            return cos;
+    }
+

...

Quoted 13 lines

xen/arch/x86/psr.c +8 -2

+    else
+    {
+        bool_t need_write = 1;
 
-        if ( ret )
+        cos = pick_avail_cos(map, cos_max, old_cos);
+        if ( cos < 0 )
         {
             spin_unlock(&info->cbm_lock);
-            return ret;
+            return cos;
+        }
+

xen/arch/x86/psr.c int psr_set_l3_cbm(struct domain *d, unsigned int socket, uint64_t cbm) · line 513

Quoted 10 lines

xen/arch/x86/psr.c +10 -0

+        /* We try to avoid writing MSR. */
+        if ( cdp_enabled )
+        {
+            if ( map[cos].code == cbm_code &&
+                 map[cos].data == cbm_data )
+                need_write = 0;
+        }
+        else
+            need_write = !(map[cos].cbm == cbm_code);
+

need_write here seems useless, you can put it in inline below.

xen/arch/x86/psr.c int psr_set_l3_cbm(struct domain *d, unsigned int socket, uint64_t cbm) · line 525

Quoted 13 lines

xen/arch/x86/psr.c +10 -1

+        if ( need_write )
+        {
+            ret = write_l3_cbm(socket, cos, cbm_code, cbm_data, cdp_enabled);
+            if ( ret )
+            {
+                spin_unlock(&info->cbm_lock);
+                return ret;
+            }
+            map[cos].code = cbm_code;
+            map[cos].data = cbm_data;
         }
-        found->cbm = cbm;
     }

Chao

He Chen

author

2015-09-28 08:29 UTC #5 lore raw

PATCH v5 3/3 tools & docs: add tools and docs support for Intel CDP

In reply to He Chen

This is the xl/xc changes to support Intel Code/Data Prioritization. CAT xl commands to set/get CBMs are extended to support CDP. Add new CDP options with CAT commands in xl interface man page. Add description of CDP in xl-psr.markdown.

Signed-off-by He Chen <[email protected]>

---
Changes in v5:
* merge tools and docs patches
* replace EINVAL with ENXIO in libxl__psr_cat_log_err_msg
* revise options parsing in psr-cat-cbm-set and invalidate passing -c and -d at the same time
* refine CDP status output codes in psr_cat_hwinfo
* adjust CBM output format in command xl psr-cat-show
* docs revision

Example of new output format for command xl psr-cat-show:

CAT-only:

Socket ID : 0
L3 Cache : 56320KB
Default CBM : 0xfffff
ID NAME CBM

    0                 Domain-0               0xfffff
    1               centos.hvm               0xfffff

CDP enabled:

Socket ID : 0
L3 Cache : 56320KB
Default CBM : 0xfffff
ID NAME CBM

    0                 Domain-0     code: 0xfffff     data: 0xfffff
    1               centos.hvm     code: 0xfffff     data: 0xfffff
---
 docs/man/xl.pod.1             | 15 ++++++++++++
 docs/misc/xl-psr.markdown     | 47 +++++++++++++++++++++++++++++++-----
 tools/libxc/include/xenctrl.h |  7 ++++--
 tools/libxc/xc_psr.c          | 17 ++++++++++++-
 tools/libxl/libxl.h           |  7 ++++++
 tools/libxl/libxl_psr.c       |  5 +++-
 tools/libxl/libxl_types.idl   |  3 +++
 tools/libxl/xl_cmdimpl.c      | 56 +++++++++++++++++++++++++++++++++----------
 tools/libxl/xl_cmdtable.c     |  2 ++
 9 files changed, 137 insertions(+), 22 deletions(-)

docs/man/xl.pod.1 +15 -0

index f22c3f3..6388351 100644
@@ -1530,6 +1530,13 @@ applications. In the Xen implementation, CAT is used to control cache allocation
 on VM basis. To enforce cache on a specific domain, just set capacity bitmasks
 (CBM) for the domain.
 
+Intel Broadwell and later server platforms also offer Code/Data Prioritization
+(CDP) for cache allocations, which support specifying code or data cache for
+applications. CDP is used on a per VM basis in the Xen implementation. To
+specify code or data CBM for the domain, CDP feature must be enabled and CBM
+type options need to be specified when setting CBM, and the type options (code
+and data) are mutually exclusive.
+
 =over 4
 
 =item B<psr-cat-cbm-set> [I<OPTIONS>] I<domain-id> I<cbm>

@@ -1545,6 +1552,14 @@ B<OPTIONS>
 
 Specify the socket to process, otherwise all sockets are processed.
 
+=item B<-c>, B<--code>
+
+Set code CBM when CDP is enabled.
+
+=item B<-d>, B<--data>
+
+Set data CBM when CDP is enabled.
+
 =back
 
 =item B<psr-cat-show> [I<domain-id>]

docs/misc/xl-psr.markdown +41 -6

78 lines not on this page · raw message

tools/libxc/include/xenctrl.h +5 -2

index 2000f12..8f4acec 100644
@@ -2794,7 +2794,9 @@ enum xc_psr_cmt_type {
 typedef enum xc_psr_cmt_type xc_psr_cmt_type;
 
 enum xc_psr_cat_type {
-    XC_PSR_CAT_L3_CBM = 1,
+    XC_PSR_CAT_L3_CBM  = 1,
+    XC_PSR_CAT_L3_CODE = 2,
+    XC_PSR_CAT_L3_DATA = 3,
 };
 typedef enum xc_psr_cat_type xc_psr_cat_type;

@@ -2820,7 +2822,8 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
                                xc_psr_cat_type type, uint32_t target,
                                uint64_t *data);
 int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
-                           uint32_t *cos_max, uint32_t *cbm_len);
+                           uint32_t *cos_max, uint32_t *cbm_len,
+                           bool *cdp_enabled);
 #endif
 
 #endif /* XENCTRL_H */

tools/libxc/xc_psr.c +16 -1

index d8b3a51..5bbe950 100644
@@ -260,6 +260,12 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
     case XC_PSR_CAT_L3_CBM:
         cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM;
         break;
+    case XC_PSR_CAT_L3_CODE:
+        cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE;
+        break;
+    case XC_PSR_CAT_L3_DATA:
+        cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA;
+        break;
     default:
         errno = EINVAL;
         return -1;

@@ -287,6 +293,12 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
     case XC_PSR_CAT_L3_CBM:
         cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM;
         break;
+    case XC_PSR_CAT_L3_CODE:
+        cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE;
+        break;
+    case XC_PSR_CAT_L3_DATA:
+        cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA;
+        break;
     default:
         errno = EINVAL;
         return -1;

@@ -306,7 +318,8 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
 }
 
 int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
-                           uint32_t *cos_max, uint32_t *cbm_len)
+                           uint32_t *cos_max, uint32_t *cbm_len,
+                           bool *cdp_enabled)
 {
     int rc;
     DECLARE_SYSCTL;

@@ -320,6 +333,8 @@ int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
     {
         *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
         *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
+        *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
+                       XEN_SYSCTL_PSR_CAT_L3_CDP;
     }
 
     return rc;

tools/libxl/libxl.h +7 -0

index 5f9047c..611e98d 100644
@@ -796,6 +796,13 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, libxl_mac *src);
  * If this is defined, the Cache Allocation Technology feature is supported.
  */
 #define LIBXL_HAVE_PSR_CAT 1
+
+/*
+ * LIBXL_HAVE_PSR_CDP
+ *
+ * If this is defined, the Code and Data Prioritization feature is supported.
+ */
+#define LIBXL_HAVE_PSR_CDP 1
 #endif
 
 /*

tools/libxl/libxl_psr.c +4 -1

index 3378239..4ad0fc8 100644
@@ -87,6 +87,9 @@ static void libxl__psr_cat_log_err_msg(libxl__gc *gc, int err)
     case EEXIST:
         msg = "The same CBM is already set to this domain";
         break;
+    case ENXIO:
+        msg = "Unable to set code or data CBM when CDP is disabled";
+        break;
 
     default:
         libxl__psr_log_err_msg(gc, err);

@@ -352,7 +355,7 @@ int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
 
     for (i = 0; i < nr_sockets; i++) {
         if (xc_psr_cat_get_l3_info(ctx->xch, i, &ptr[i].cos_max,
-                                                &ptr[i].cbm_len)) {
+                                   &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
             libxl__psr_cat_log_err_msg(gc, errno);
             rc = ERROR_FAIL;
             free(ptr);

tools/libxl/libxl_types.idl +3 -0

index ef346e7..fa017ad 100644
@@ -787,9 +787,12 @@ libxl_psr_cmt_type = Enumeration("psr_cmt_type", [
 libxl_psr_cbm_type = Enumeration("psr_cbm_type", [
     (0, "UNKNOWN"),
     (1, "L3_CBM"),
+    (2, "L3_CODE"),
+    (3, "L3_DATA"),
     ])
 
 libxl_psr_cat_info = Struct("psr_cat_info", [
     ("cos_max", uint32),
     ("cbm_len", uint32),
+    ("cdp_enabled", bool),
     ])

tools/libxl/xl_cmdimpl.c +44 ~~-12~~

121 lines not on this page · raw message

tools/libxl/xl_cmdtable.c +2 -0

index 0071f12..fdc1ac6 100644
@@ -543,6 +543,8 @@ struct cmd_spec cmd_table[] = {
       "Set cache capacity bitmasks(CBM) for a domain",
       "[options] <Domain> <CBM>",
       "-s <socket>       Specify the socket to process, otherwise all sockets are processed\n"
+      "-c                Set code CBM if CDP is supported\n"
+      "-d                Set data CBM if CDP is supported\n"
     },
     { "psr-cat-show",
       &main_psr_cat_show, 0, 1,

Signature · 1 line

1.9.1

Ian Campbell

2015-09-29 13:47 UTC #6 lore raw

Re: [PATCH v5 3/3] tools & docs: add tools and docs support for Intel CDP

Quoted 27 lines · He Chen

This is the xl/xc changes to support Intel Code/Data Prioritization. CAT xl commands to set/get CBMs are extended to support CDP. Add new CDP options with CAT commands in xl interface man page. Add description of CDP in xl-psr.markdown.
Signed-off-by He Chen <[email protected]>
---
Changes in v5:
* merge tools and docs patches
* replace EINVAL with ENXIO in libxl__psr_cat_log_err_msg
* revise options parsing in psr-cat-cbm-set and invalidate passing -c and -d at the same time
* refine CDP status output codes in psr_cat_hwinfo
* adjust CBM output format in command xl psr-cat-show
* docs revision
Example of new output format for command xl psr-cat-show:
CAT-only:
Socket ID : 0
L3 Cache : 56320KB
Default CBM : 0xfffff
ID NAME CBM
    0                 Domain-0               0xfffff
    1               centos.hvm               0xfffff

Quoted 8 lines

CDP enabled:
Socket ID : 0
L3 Cache : 56320KB
Default CBM : 0xfffff
ID NAME CBM
    0                 Domain-0     code: 0xfffff     data: 0xfffff
    1               centos.hvm     code: 0xfffff     data: 0xfffff

The CBM column header seems a bit out of place.

Perhaps have separate headings, e.g. "CBM (code)" and "CBD (data)"?

docs/misc/xl-psr.markdown +0 -0

Quoted 10 lines

index 3545912..0527211 100644
docs/misc/xl-psr.markdown +0 -0
@@ -14,7 +14,7 @@ tracks cache utilization of memory accesses according
to the RMID and reports
monitored data via a counter register.
For more detailed information please refer to Intel SDM chapter
-"17.14 - Platform Shared Resource Monitoring: Cache Monitoring Technology". +"Platform Shared Resource Monitoring: Cache Monitoring Technology".

I think these will clash with Cheng Pao's PSR updates, which have already been applied.

Quoted 2 lines

+Code/Data Prioritization (CDP) Technology is an extension of CAT, which is

Other bits of documentation added here seem to suggest it is actually an extension of CBM, is that right?

tools/libxc/include/xenctrl.h enum xc_psr_cmt_type { · line 2799

Quoted 13 lines

tools/libxc/include/xenctrl.h +3 -1

index 2000f12..8f4acec 100644
@@ -2794,7 +2794,9 @@ enum xc_psr_cmt_type {
 typedef enum xc_psr_cmt_type xc_psr_cmt_type;
 
 enum xc_psr_cat_type {
-    XC_PSR_CAT_L3_CBM = 1,
+    XC_PSR_CAT_L3_CBM  = 1,
+    XC_PSR_CAT_L3_CODE = 2,
+    XC_PSR_CAT_L3_DATA = 3,

If CDP is an extension of CBM, then would CBM_CODE + CBM_DATA be better names (more important in the libxl API than here)

tools/libxc/include/xenctrl.h enum xc_psr_cmt_type { · line 2801

Quoted 2 lines

tools/libxc/include/xenctrl.h +2 -0

+    if (opt_data && opt_code) {
+        fprintf(stderr, "Invalid to pass both -c and -d at the same time\n");

To be correct this would need to say "It is invalid..." but a more natural sounding (to me) way to express this would be

"Cannot handle -c and -d at the same time"

I think.

Chao Peng

2015-09-30 02:50 UTC #7 lore raw

Re: [PATCH v5 3/3] tools & docs: add tools and docs support for Intel CDP

Quoted 6 lines · Ian Campbell

Quoted 2 lines · He Chen
-"17.14 - Platform Shared Resource Monitoring: Cache Monitoring Technology". +"Platform Shared Resource Monitoring: Cache Monitoring Technology".
I think these will clash with Cheng Pao's PSR updates, which have already been applied.

Yes, this can be eliminated in this series.

Quoted 6 lines

Quoted 3 lines
+Code/Data Prioritization (CDP) Technology is an extension of CAT, which is
Other bits of documentation added here seem to suggest it is actually an extension of CBM, is that right?

Yes, from technical perspective, CDP is an extension on CAT. But even though, I'd really like it being put in a separated section, making it easy for user to find it out. The new section can of course refers to CAT section, just like what we did for MBM (which is an extension of CMT).

Quoted 9 lines

tools/libxc/include/xenctrl.h +0 -0

index 2000f12..8f4acec 100644
@@ -2794,7 +2794,9 @@ enum xc_psr_cmt_type {
 typedef enum xc_psr_cmt_type xc_psr_cmt_type;

Quoted 8 lines

tools/libxc/include/xenctrl.h enum xc_psr_cmt_type { · line 2799
Quoted 5 lines
tools/libxc/include/xenctrl.h +3 -1
 enum xc_psr_cat_type {
-    XC_PSR_CAT_L3_CBM = 1,
+    XC_PSR_CAT_L3_CBM  = 1,
+    XC_PSR_CAT_L3_CODE = 2,
+    XC_PSR_CAT_L3_DATA = 3,
If CDP is an extension of CBM, then would CBM_CODE + CBM_DATA be better names (more important in the libxl API than here)

Agreed.

Chao