detect and initialize CDP (Code/Data Prioritization) feature

Changes in v8:
- x86:
* x86/psr part is not included in this version since it had been merged to upstream
- tools & docs:
* fix per socekt CDP status show bug
* fix word missing in xl-psr.markdown
* coding style

Changes in v7:
- x86:
* amend function find_cos and pick_avail_cos to ignore reference count for COS0
* write CDP data mask (mask1) before turning CDP on
* check CDP status before clear enabled bit in function cat_cpu_fini
* code style
- tools & docs:
* LIBXL_PSR_CBM_TYPE_L3_CODE (DATA) => LIBXL_PSR_CBM_TYPE_L3_CBM_CODE (DATA)
* refactor psr_cat_print_one_domain and add a helper to print different type CBM
* fix docs wrap

Changes in v6:
- x86:
* remove unnecessary parameter in cdp_is_enabled function
* remove variable need_write and restruct code in psr_set_l3_cbm
- tools & docs:
* separate CBM headings in the output of xl psr-cat-show
* revert the numbers of SDM chapter in xl-psr.markdown
* XC_PSR_CAT_L3_CODE (DATA) => XC_PSR_CAT_L3_CBM_CODE (DATA)
* correct error message of passing -c and -d at the same time

Changes in v5:
- x86: address Andrew's and Jan's coments.
- tools: refine options parsing in psr-cat-cbm-set and invalidate passing -c and -d at the same time
- merge tools and docs patches.
- code style

Changes in v4:
- x86:
* remove union member name in struct psr_cat_cbm (suggested by Jan)
* fix log info of CAT & CDP (suggested by Chao & Jan)
* add a helper cdp_is_enabled to tell the status of CDP and CDP initialize failed is considered (Jan's comment)
* XEN_SYSCTL_INTERFACE_VERSION 0x0000000C -> 0x0000000D (suggested by Jan)
* refine CBM type check logic in get/set CBM function (suggested by Jan)
* loop optimization in function find_cos (suggested by Jan)
- tools: address Chao's comments.
- docs: address Chao's comments.
- code style

Changes in v3:
- x86: remove redundant CDP field in cat_socket_enable (suggested by Chao)
- tools: simplify CBM setting function in tools (suggested by Jan)
- docs: add boot parameter description (suggested by Chao & Ian)
- code style

Changes in v2:
- x86: enable CDP by boot parameter instead of enabling/disabling CDP at runtime (suggested by Andrew)
- tools: remove psr-cat-cdp-enable/disable xl commands
- code style

Code/Data Prioritization(CDP) is offered in Intel Broadwell and later server platforms, which is an extension of CAT. CDP enables isolation and separate prioritization of code and data fetches to the L3 cache in a software configurable manner, which can enable workload prioritization and tuning of cache capacity to the characteristics of the workload. CDP extends Cache Allocation Technology (CAT) by providing separate code and data capacity bit masks(CBM) per Class of Service (COS). CDP is used on VM basis in the Xen implementation.

More information about CDP, please refer to Intel SDM, Volumn 3, section 17.16
http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-manual-325462.pdf

This patch series enables CDP feature in Xen based on CAT code, and extends CBM operation functions to support CDP. For all the changes, please see in each patch.

This v8 patch has been tested on Intel Broadwell server platform.

To make this patchset better, any comment or suggestion is welcomed, I would really appreciate it.

Thanks.

He Chen (1):
tools & docs: add tools and docs support for Intel CDP

 docs/man/xl.pod.1             | 15 ++++++++++
 docs/misc/xl-psr.markdown     | 53 ++++++++++++++++++++++++++++++++++
 tools/libxc/include/xenctrl.h |  7 +++--
 tools/libxc/xc_psr.c          | 17 ++++++++++-
 tools/libxl/libxl.h           |  7 +++++
 tools/libxl/libxl_psr.c       |  5 +++-
 tools/libxl/libxl_types.idl   |  3 ++
 tools/libxl/xl_cmdimpl.c      | 67 +++++++++++++++++++++++++++++++++++--------
 tools/libxl/xl_cmdtable.c     |  2 ++
 9 files changed, 160 insertions(+), 16 deletions(-)

1.9.1

This is the xl/xc changes to support Intel Code/Data Prioritization. CAT xl commands to set/get CBMs are extended to support CDP. Add new CDP options with CAT commands in xl interface man page. Add description of CDP in xl-psr.markdown.

---
Changes in v8:
* fix per socekt CDP status show bug
* fix word missing in xl-psr.markdown
* coding style

Changes in v7:
* LIBXL_PSR_CBM_TYPE_L3_CODE (DATA) => LIBXL_PSR_CBM_TYPE_L3_CBM_CODE (DATA)
* refactor psr_cat_print_one_domain and add a helper to print different type CBM
* fix docs wrap

Changes in v6:
* separate CBM headings in the output of xl psr-cat-show
* revert the numbers of SDM chapter in xl-psr.markdown
* XC_PSR_CAT_L3_CODE (DATA) => XC_PSR_CAT_L3_CBM_CODE (DATA)
* correct error message of passing -c and -d at the same time

Changes in v5:
* merge tools and docs patches
* replace EINVAL with ENXIO in libxl__psr_cat_log_err_msg
* revise options parsing in psr-cat-cbm-set and invalidate passing -c and -d at the same time
* refine CDP status output codes in psr_cat_hwinfo
* adjust CBM output format in command xl psr-cat-show
* docs revision

Example of new output format for command xl psr-cat-show:

*** CAT-only ***

Socket ID : 0
L3 Cache : 56320KB
Default CBM : 0xfffff
ID NAME CBM

    0                 Domain-0         0xfffff
    1               centos.hvm         0xfffff

*** CDP enabled ***

Socket ID : 0
L3 Cache : 56320KB
Default CBM : 0xfffff
ID NAME CBM (code) CBM (data)

    0                 Domain-0         0xfffff        0xfffff
    1               centos.hvm         0xfffff        0xfffff
---
 docs/man/xl.pod.1             | 15 ++++++++++
 docs/misc/xl-psr.markdown     | 53 ++++++++++++++++++++++++++++++++++
 tools/libxc/include/xenctrl.h |  7 +++--
 tools/libxc/xc_psr.c          | 17 ++++++++++-
 tools/libxl/libxl.h           |  7 +++++
 tools/libxl/libxl_psr.c       |  5 +++-
 tools/libxl/libxl_types.idl   |  3 ++
 tools/libxl/xl_cmdimpl.c      | 67 +++++++++++++++++++++++++++++++++++--------
 tools/libxl/xl_cmdtable.c     |  2 ++
 9 files changed, 160 insertions(+), 16 deletions(-)

index d0cd612..4279c7c 100644
@@ -1518,6 +1518,13 @@ applications. In the Xen implementation, CAT is used to control cache allocation
 on VM basis. To enforce cache on a specific domain, just set capacity bitmasks
 (CBM) for the domain.
 
+Intel Broadwell and later server platforms also offer Code/Data Prioritization
+(CDP) for cache allocations, which support specifying code or data cache for
+applications. CDP is used on a per VM basis in the Xen implementation. To
+specify code or data CBM for the domain, CDP feature must be enabled and CBM
+type options need to be specified when setting CBM, and the type options (code
+and data) are mutually exclusive.
+
 =over 4
 
 =item B<psr-cat-cbm-set> [I<OPTIONS>] I<domain-id> I<cbm>

@@ -1533,6 +1540,14 @@ B<OPTIONS>
 
 Specify the socket to process, otherwise all sockets are processed.
 
+=item B<-c>, B<--code>
+
+Set code CBM when CDP is enabled.
+
+=item B<-d>, B<--data>
+
+Set data CBM when CDP is enabled.
+
 =back
 
 =item B<psr-cat-show> [I<domain-id>]

index 3bfa00b..2fec1fb 100644
@@ -2827,7 +2827,9 @@ enum xc_psr_cmt_type {
 typedef enum xc_psr_cmt_type xc_psr_cmt_type;
 
 enum xc_psr_cat_type {
-    XC_PSR_CAT_L3_CBM = 1,
+    XC_PSR_CAT_L3_CBM      = 1,
+    XC_PSR_CAT_L3_CBM_CODE = 2,
+    XC_PSR_CAT_L3_CBM_DATA = 3,
 };
 typedef enum xc_psr_cat_type xc_psr_cat_type;
 

@@ -2853,7 +2855,8 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
                                xc_psr_cat_type type, uint32_t target,
                                uint64_t *data);
 int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
-                           uint32_t *cos_max, uint32_t *cbm_len);
+                           uint32_t *cos_max, uint32_t *cbm_len,
+                           bool *cdp_enabled);
 #endif
 
 #endif /* XENCTRL_H */

index d8b3a51..43b3286 100644
@@ -260,6 +260,12 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,
     case XC_PSR_CAT_L3_CBM:
         cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM;
         break;
+    case XC_PSR_CAT_L3_CBM_CODE:
+        cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE;
+        break;
+    case XC_PSR_CAT_L3_CBM_DATA:
+        cmd = XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA;
+        break;
     default:
         errno = EINVAL;
         return -1;

@@ -287,6 +293,12 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
     case XC_PSR_CAT_L3_CBM:
         cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM;
         break;
+    case XC_PSR_CAT_L3_CBM_CODE:
+        cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE;
+        break;
+    case XC_PSR_CAT_L3_CBM_DATA:
+        cmd = XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA;
+        break;
     default:
         errno = EINVAL;
         return -1;

@@ -306,7 +318,8 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,
 }
 
 int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
-                           uint32_t *cos_max, uint32_t *cbm_len)
+                           uint32_t *cos_max, uint32_t *cbm_len,
+                           bool *cdp_enabled)
 {
     int rc;
     DECLARE_SYSCTL;

@@ -320,6 +333,8 @@ int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket,
     {
         *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max;
         *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len;
+        *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags &
+                       XEN_SYSCTL_PSR_CAT_L3_CDP;
     }
 
     return rc;

index 615b1de..168fedd 100644
@@ -803,6 +803,13 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, libxl_mac *src);
  * If this is defined, the Cache Allocation Technology feature is supported.
  */
 #define LIBXL_HAVE_PSR_CAT 1
+
+/*
+ * LIBXL_HAVE_PSR_CDP
+ *
+ * If this is defined, the Code and Data Prioritization feature is supported.
+ */
+#define LIBXL_HAVE_PSR_CDP 1
 #endif
 
 /*

index 000d748..3d0dc61 100644
@@ -87,6 +87,9 @@ static void libxl__psr_cat_log_err_msg(libxl__gc *gc, int err)
     case EEXIST:
         msg = "The same CBM is already set to this domain";
         break;
+    case ENXIO:
+        msg = "Unable to set code or data CBM when CDP is disabled";
+        break;
 
     default:
         libxl__psr_log_err_msg(gc, err);

@@ -363,7 +366,7 @@ int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,
     libxl_for_each_set_bit(socketid, socketmap) {
         ptr[i].id = socketid;
         if (xc_psr_cat_get_l3_info(ctx->xch, socketid, &ptr[i].cos_max,
-                                   &ptr[i].cbm_len)) {
+                                   &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {
             libxl__psr_cat_log_err_msg(gc, errno);
             rc = ERROR_FAIL;
             free(ptr);

index d6ef9a2..4d78f86 100644
@@ -797,10 +797,13 @@ libxl_psr_cmt_type = Enumeration("psr_cmt_type", [
 libxl_psr_cbm_type = Enumeration("psr_cbm_type", [
     (0, "UNKNOWN"),
     (1, "L3_CBM"),
+    (2, "L3_CBM_CODE"),
+    (3, "L3_CBM_DATA"),
     ])
 
 libxl_psr_cat_info = Struct("psr_cat_info", [
     ("id", uint32),
     ("cos_max", uint32),
     ("cbm_len", uint32),
+    ("cdp_enabled", bool),
     ])

index 0071f12..fdc1ac6 100644
@@ -543,6 +543,8 @@ struct cmd_spec cmd_table[] = {
       "Set cache capacity bitmasks(CBM) for a domain",
       "[options] <Domain> <CBM>",
       "-s <socket>       Specify the socket to process, otherwise all sockets are processed\n"
+      "-c                Set code CBM if CDP is supported\n"
+      "-d                Set data CBM if CDP is supported\n"
     },
     { "psr-cat-show",
       &main_psr_cat_show, 0, 1,

1.9.1

Applied, thanks.

Ian.