detect and initialize CDP (Code/Data Prioritization) feature
v6 cover detect and initialize CDP (Code/Data Prioritization) feature
Changes in v6:
- x86:
* remove unnecessary parameter in cdp_is_enabled function
* remove variable need_write and restruct code in psr_set_l3_cbm
- tools & docs:
* separate CBM headings in the output of xl psr-cat-show
* revert the numbers of SDM chapter in xl-psr.markdown
* XC_PSR_CAT_L3_CODE (DATA) => XC_PSR_CAT_L3_CBM_CODE (DATA)
* correct error message of passing -c and -d at the same time
Changes in v5:
- x86: address Andrew's and Jan's coments.
- tools: refine options parsing in psr-cat-cbm-set and invalidate passing -c and -d at the same time
- merge tools and docs patches.
- code style
Changes in v4:
- x86:
* remove union member name in struct psr_cat_cbm (suggested by Jan)
* fix log info of CAT & CDP (suggested by Chao & Jan)
* add a helper cdp_is_enabled to tell the status of CDP and CDP initialize failed is considered (Jan's comment)
* XEN_SYSCTL_INTERFACE_VERSION 0x0000000C -> 0x0000000D (suggested by Jan)
* refine CBM type check logic in get/set CBM function (suggested by Jan)
* loop optimization in function find_cos (suggested by Jan)
- tools: address Chao's comments.
- docs: address Chao's comments.
- code style
Changes in v3:
- x86: remove redundant CDP field in cat_socket_enable (suggested by Chao)
- tools: simplify CBM setting function in tools (suggested by Jan)
- docs: add boot parameter description (suggested by Chao & Ian)
- code style
Changes in v2:
- x86: enable CDP by boot parameter instead of enabling/disabling CDP at runtime (suggested by Andrew)
- tools: remove psr-cat-cdp-enable/disable xl commands
- code style
Code/Data Prioritization(CDP) is offered in Intel Broadwell and later server platforms, which is an extension of CAT. CDP enables isolation and separate prioritization of code and data fetches to the L3 cache in a software configurable manner, which can enable workload prioritization and tuning of cache capacity to the characteristics of the workload. CDP extends Cache Allocation Technology (CAT) by providing separate code and data capacity bit masks(CBM) per Class of Service (COS). CDP is used on VM basis in the Xen implementation.
More information about CDP, please refer to Intel SDM, Volumn 3, section 17.16
http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-manual-325462.pdf
This patch series enables CDP feature in Xen based on CAT code, and extends CBM operation functions to support CDP. For all the changes, please see in each patch.
This v6 patchset has been tested on Intel Broadwell server platform.
To make this patchset better, any comment or suggestion is welcomed, I would really appreciate it.
Thanks.
He Chen (3):
x86: Support enable CDP by boot parameter and add get CDP status
x86: add domctl cmd to set/get CDP code/data CBM tools & docs: add tools and docs support for Intel CDP
docs/man/xl.pod.1 | 15 +++ docs/misc/xen-command-line.markdown | 11 +- docs/misc/xl-psr.markdown | 53 +++++++++ tools/libxc/include/xenctrl.h | 7 +- tools/libxc/xc_psr.c | 17 ++- tools/libxl/libxl.h | 7 ++ tools/libxl/libxl_psr.c | 5 +- tools/libxl/libxl_types.idl | 3 + tools/libxl/xl_cmdimpl.c | 58 +++++++-- tools/libxl/xl_cmdtable.c | 2 + xen/arch/x86/domctl.c | 32 ++++- xen/arch/x86/psr.c | 229 ++++++++++++++++++++++++++++-------- xen/arch/x86/sysctl.c | 5 +- xen/include/asm-x86/msr-index.h | 3 + xen/include/asm-x86/psr.h | 20 +++- xen/include/public/domctl.h | 4 + xen/include/public/sysctl.h | 4 +- 17 files changed, 404 insertions(+), 71 deletions(-)
Signature · 1 line
1.9.1
PATCH v6 1/3 x86: Support enable CDP by boot parameter and add get CDP status
Add boot parameter psr=cdp to enable CDP at boot time. Intel Code/Data Prioritization (CDP) feature is based on CAT. Note that cos_max would be half when CDP is on. struct psr_cat_cbm is extended to support CDP operation. Extend psr_get_cat_l3_info sysctl to get CDP status.
---
Changes in v6:
* remove unnecessary parameter in cdp_is_enabled
Changes in v5:
* remove unnecessary u in psr_cat_cbm structure
* revert write_l3_cbm and put the modification to next patch
* remove duplicate PSR_CAT_FLAG_L3_CDP
--- docs/misc/xen-command-line.markdown | 11 +++++++-- xen/arch/x86/psr.c | 48 +++++++++++++++++++++++++++++++++---- xen/arch/x86/sysctl.c | 5 ++-- xen/include/asm-x86/msr-index.h | 3 +++ xen/include/asm-x86/psr.h | 8 ++++++- xen/include/public/sysctl.h | 4 +++- 6 files changed, 69 insertions(+), 10 deletions(-)
docs/misc/xen-command-line.markdown +9 -2
@@ -1174,9 +1174,9 @@ This option can be specified more than once (up to 8 times at present). > `= <integer>` ### psr (Intel) -> `= List of ( cmt:<boolean> | rmid_max:<integer> | cat:<boolean> | cos_max:<integer> )` +> `= List of ( cmt:<boolean> | rmid_max:<integer> | cat:<boolean> | cos_max:<integer> | cdp:<boolean> )` -> Default: `psr=cmt:0,rmid_max:255,cat:0,cos_max:255` +> Default: `psr=cmt:0,rmid_max:255,cat:0,cos_max:255,cdp:0` Platform Shared Resource(PSR) Services. Intel Haswell and later server platforms offer information about the sharing of resources.
@@ -1206,6 +1206,13 @@ The following resources are available: the cache allocation. * `cat` instructs Xen to enable/disable Cache Allocation Technology. * `cos_max` indicates the max value for COS ID. +* Code and Data Prioritization Technology (Broadwell and later). Information + regarding the code cache and the data cache allocation. CDP is based on CAT. + * `cdp` instructs Xen to enable/disable Code and Data Prioritization. Note + that `cos_max` of CDP is a little different from `cos_max` of CAT. With + CDP, one COS will corespond two CBMs other than one with CAT, due to the + sum of CBMs is fixed, that means actual `cos_max` in use will automatically + reduce to half when CDP is enabled. ### reboot > `= t[riple] | k[bd] | a[cpi] | p[ci] | P[ower] | e[fi] | n[o] [, [w]arm | [c]old]`
xen/arch/x86/psr.c +44 -4
105 lines not on this page · raw message
xen/arch/x86/sysctl.c +3 -2
@@ -178,12 +178,13 @@ long arch_do_sysctl(case XEN_SYSCTL_PSR_CAT_get_l3_info: ret =psr_get_cat_l3_info (sysctl->u.psr_cat_op.target, &sysctl->u.psr_cat_op.u.l3_info.cbm_len, - &sysctl->u.psr_cat_op.u.l3_info.cos_max); + &sysctl->u.psr_cat_op.u.l3_info.cos_max, + &sysctl->u.psr_cat_op.u.l3_info.flags);if ( !ret &&__copy_field_to_guest (u_sysctl, sysctl, u.psr_cat_op) ) ret = -EFAULT ; -break ; +default : ret = -EOPNOTSUPP ;break ;
xen/include/asm-x86/msr-index.h +3 -0
@@ -328,7 +328,10 @@#define MSR_IA32_CMT_EVTSEL 0x00000c8d#define MSR_IA32_CMT_CTR 0x00000c8e#define MSR_IA32_PSR_ASSOC 0x00000c8f +#define MSR_IA32_PSR_L3_QOS_CFG 0x00000c81#define MSR_IA32_PSR_L3_MASK (n) (0x00000c90 + (n)) +#define MSR_IA32_PSR_L3_MASK_CODE (n) (0x00000c90 + (n) * 2 + 1) +#define MSR_IA32_PSR_L3_MASK_DATA (n) (0x00000c90 + (n) * 2)/* Intel Model 6 */ #define MSR_P6_PERFCTR (n) (0x000000c1 + (n))
xen/include/asm-x86/psr.h +7 -1
@@ -27,6 +27,12 @@/* L3 Monitoring Features */ #define PSR_CMT_L3_OCCUPANCY 0x1 +/* CDP Capability */ +#define PSR_CAT_CDP_CAPABILITY (1u << 2) + +/* L3 CDP Enable bit*/ +#define PSR_L3_QOS_CDP_ENABLE_BIT 0x0 +struct psr_cmt_l3 {unsigned int features;unsigned int upscaling_factor;
@@ -52,7 +58,7 @@ void psr_free_rmid(struct domain *d);void psr_ctxt_switch_to (struct domain *d);int psr_get_cat_l3_info (unsigned int socket,uint32_t *cbm_len, -uint32_t *cos_max); +uint32_t *cos_max,uint32_t *flags);int psr_get_l3_cbm (struct domain *d,unsigned int socket,uint64_t *cbm);int psr_set_l3_cbm (struct domain *d,unsigned int socket,uint64_t cbm);
xen/include/public/sysctl.h +3 -1
@@ -36,7 +36,7 @@#include "physdev.h" #include "tmem.h" -#define XEN_SYSCTL_INTERFACE_VERSION 0x0000000C +#define XEN_SYSCTL_INTERFACE_VERSION 0x0000000D/* * Read console content from Xen buffer ring.
@@ -705,6 +705,8 @@ struct xen_sysctl_psr_cat_op {struct {uint32_t cbm_len;/* OUT: CBM length */ uint32_t cos_max;/* OUT: Maximum COS */ +#define XEN_SYSCTL_PSR_CAT_L3_CDP (1u << 0) +uint32_t flags;/* OUT: CAT flags */ } l3_info; } u; };
Signature · 1 line
1.9.1
Re: [PATCH v6 1/3] x86: Support enable CDP by boot parameter and add get CDP status
xen/arch/x86/psr.c static void cat_cpu_init(void) · line 528
Quoted 11 lines
xen/arch/x86/psr.c +11
-0+if ( (ecx &PSR_CAT_CDP_CAPABILITY ) && (opt_psr &PSR_CDP ) && + cdp_socket_enable && !test_bit (socket, cdp_socket_enable) ) + { +rdmsrl (MSR_IA32_PSR_L3_QOS_CFG , val); +wrmsrl (MSR_IA32_PSR_L3_QOS_CFG , val | (1 <<PSR_L3_QOS_CDP_ENABLE_BIT )); + + info->cos_to_cbm[0 ].code = (1ull << info->cbm_len) -1 ; + info->cos_to_cbm[0 ].data = (1ull << info->cbm_len) -1 ; + +/* We only write mask1 since mask0 is always all ones by default. */ +wrmsrl (MSR_IA32_PSR_L3_MASK (1 ), (1ull << info->cbm_len) -1 );
Although I think it maybe not a big deal here, I'd suggest doing this before you actually turning CDP on (writing MSR_IA32_PSR_L3_QOS_CFG above) as strictly speaking before we enabling CDP the data mask(mask1) for cos0 should already be initialised.
Quoted 13 lines
xen/arch/x86/psr.c +9
-0+ +/* Cut half of cos_max when CDP is enabled. */ + info->cos_max >>=1 ; + +set_bit (socket, cdp_socket_enable); + } +printk (XENLOG_INFO "CAT: enabled on socket %u, cos_max:%u, cbm_len:%u, CDP:%s\n" , + socket, info->cos_max, info->cbm_len, +cdp_is_enabled (socket) ?"on" :"off" ); } }
xen/arch/x86/psr.c static void cat_cpu_fini(unsigned int cpu) · line 554
Quoted 4 lines
xen/arch/x86/psr.c +1
-0@@ -513,6 +551,7 @@ static void cat_cpu_fini(unsigned int cpu)xfree (info->cos_to_cbm); info->cos_to_cbm =NULL ; } +clear_bit (socket, cdp_socket_enable);
Is it possible for cdp_socket_enable to be NULL? e.g. in the case when CDP failed to initilize. You can check if it works for cat-only mode.
Chao
Quoted 2 lines
xen/arch/x86/psr.c +0
-0clear_bit (socket, cat_socket_enable); }
PATCH v6 2/3 x86: add domctl cmd to set/get CDP code/data CBM
In reply to He ChenCDP extends CAT and provides the capacity to control L3 code & data cache. With CDP, one COS corresponds to two CMBs(code & data). cbm_type is added to distinguish different CBM operations. Besides, new domctl cmds are introdunced to support set/get CDP CBM. Some CAT functions to operation CBMs are extended to support CDP.
---
Changes in v6:
* remove variable need_write and restruct code in psr_set_l3_cbm
* remove redundant type == PSR_CBM_TYPE_L3 in psr_get_l3_cbm
Changes in v5:
* replace -EINVAL with -ENXIO when setting code/data CBM on CDP disabled.
--- xen/arch/x86/domctl.c | 32 +++++++- xen/arch/x86/psr.c | 181 ++++++++++++++++++++++++++++++++++---------- xen/include/asm-x86/psr.h | 12 ++- xen/include/public/domctl.h | 4 + 4 files changed, 183 insertions(+), 46 deletions(-)
xen/arch/x86/domctl.c +30 -2
@@ -1168,12 +1168,40 @@ long arch_do_domctl( {case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM : ret =psr_set_l3_cbm (d, domctl->u.psr_cat_op.target, - domctl->u.psr_cat_op.data); + domctl->u.psr_cat_op.data, +PSR_CBM_TYPE_L3 ); +break ; + +case XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE : + ret =psr_set_l3_cbm (d, domctl->u.psr_cat_op.target, + domctl->u.psr_cat_op.data, +PSR_CBM_TYPE_L3_CODE ); +break ; + +case XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA : + ret =psr_set_l3_cbm (d, domctl->u.psr_cat_op.target, + domctl->u.psr_cat_op.data, +PSR_CBM_TYPE_L3_DATA );break ;case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM : ret =psr_get_l3_cbm (d, domctl->u.psr_cat_op.target, - &domctl->u.psr_cat_op.data); + &domctl->u.psr_cat_op.data, +PSR_CBM_TYPE_L3 ); + copyback =1 ; +break ; + +case XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE : + ret =psr_get_l3_cbm (d, domctl->u.psr_cat_op.target, + &domctl->u.psr_cat_op.data, +PSR_CBM_TYPE_L3_CODE ); + copyback =1 ; +break ; + +case XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA : + ret =psr_get_l3_cbm (d, domctl->u.psr_cat_op.target, + &domctl->u.psr_cat_op.data, +PSR_CBM_TYPE_L3_DATA ); copyback =1 ;break ;
xen/arch/x86/psr.c +139 -42
235 lines not on this page · raw message
xen/include/asm-x86/psr.h +10 -2
@@ -46,6 +46,12 @@ struct psr_cmt {struct psr_cmt_l3 l3; }; +enum cbm_type { +PSR_CBM_TYPE_L3 , +PSR_CBM_TYPE_L3_CODE , +PSR_CBM_TYPE_L3_DATA , +}; +extern struct psr_cmt *psr_cmt;static inline bool_t psr_cmt_enabled (void )
@@ -59,8 +65,10 @@ void psr_ctxt_switch_to(struct domain *d);int psr_get_cat_l3_info (unsigned int socket,uint32_t *cbm_len,uint32_t *cos_max,uint32_t *flags); -int psr_get_l3_cbm (struct domain *d,unsigned int socket,uint64_t *cbm); -int psr_set_l3_cbm (struct domain *d,unsigned int socket,uint64_t cbm); +int psr_get_l3_cbm (struct domain *d,unsigned int socket, +uint64_t *cbm,enum cbm_type type); +int psr_set_l3_cbm (struct domain *d,unsigned int socket, +uint64_t cbm,enum cbm_type type);int psr_domain_init (struct domain *d);void psr_domain_free (struct domain *d);
xen/include/public/domctl.h +4 -0
@@ -1057,6 +1057,10 @@ DEFINE_XEN_GUEST_HANDLE(xen_domctl_monitor_op_t);struct xen_domctl_psr_cat_op {#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM 0#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM 1 +#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE 2 +#define XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA 3 +#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE 4 +#define XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA 5uint32_t cmd;/* IN: XEN_DOMCTL_PSR_CAT_OP_* */ uint32_t target;/* IN */ uint64_t data;/* IN/OUT */
Signature · 1 line
1.9.1
Re: [PATCH v6 2/3] x86: add domctl cmd to set/get CDP code/data CBM
Quoted 6 lines · He Chen
CDP extends CAT and provides the capacity to control L3 code & data cache. With CDP, one COS corresponds to two
CMBs(code & data).cbm_typeis added to distinguish different CBM operations. Besides, new domctl cmds are introdunced to support set/get CDP CBM. Some CAT functions to operation CBMs are extended to support CDP.
Quoted 6 lines
Signed-off-by He Chen <[email protected]>Reviewed-by Andrew Cooper <[email protected]>---
Changes in v6:
* remove variableneed_writeand restruct code inpsr_set_l3_cbm
* remove redundant type ==PSR_CBM_TYPE_L3inpsr_get_l3_cbm
Looks good to me. Just several coding style issues. With that fixed:
xen/arch/x86/psr.c static int write_l3_cbm(unsigned int socket, unsigned int cos, uint64_t cbm) · line 410
Quoted 8 lines
xen/arch/x86/psr.c +8
-0+static int find_cos (struct psr_cat_cbm *map,unsigned int cos_max, +uint64_t cbm_code,uint64_t cbm_data,bool_t cdp_enabled) +{ +unsigned int cos; + +for ( cos =0 ; cos <= cos_max; cos++ ) + { + if( map[cos].ref &&
^ space
xen/arch/x86/psr.c static int write_l3_cbm(unsigned int socket, unsigned int cos, uint64_t cbm) · line 413
Quoted 3 lines
xen/arch/x86/psr.c +3
-0+ ((!cdp_enabled && map[cos].cbm == cbm_code) || + (cdp_enabled && map[cos].code == cbm_code && + map[cos].data == cbm_data)))
^space
Quoted 12 lines
xen/arch/x86/psr.c +7
-0+return cos; + } + +return -ENOENT ; +}@@ -371,53 +450,71 @@ int psr_set_l3_cbm(struct domain *d, unsigned int socket, uint64_t cbm)if ( !psr_check_cbm (info->cbm_len, cbm) )return -EINVAL ; +if ( !cdp_enabled && (type ==PSR_CBM_TYPE_L3_CODE || + type ==PSR_CBM_TYPE_L3_DATA ) )
xen/arch/x86/psr.c int psr_set_l3_cbm(struct domain *d, unsigned int socket, uint64_t cbm) · line 461
Quoted 10 lines
xen/arch/x86/psr.c +4
-3+return -ENXIO ; + + cos_max = info->cos_max; old_cos = d->arch.psr_cos_ids[socket]; map = info->cos_to_cbm; -spin_lock (&info->cbm_lock); - -for ( cos =0 ; cos <= info->cos_max; cos++ ) + switch( type )
^ space
xen/arch/x86/psr.c int psr_set_l3_cbm(struct domain *d, unsigned int socket, uint64_t cbm) · line 504
Quoted 5 lines
xen/arch/x86/psr.c +5
-0+ +/* We try to avoid writing MSR. */ +if ( (cdp_enabled && + (map[cos].code != cbm_code || map[cos].data != cbm_data)) || + (!cdp_enabled && map[cos].cbm != cbm_code))
^ space
Quoted 1 line
(quoted diff) +1
-0+ {
PATCH v6 3/3 tools & docs: add tools and docs support for Intel CDP
In reply to He ChenThis is the xl/xc changes to support Intel Code/Data Prioritization. CAT xl commands to set/get CBMs are extended to support CDP. Add new CDP options with CAT commands in xl interface man page. Add description of CDP in xl-psr.markdown.
---
Changes in v6:
* separate CBM headings in the output of xl psr-cat-show
* revert the numbers of SDM chapter in xl-psr.markdown
* XC_PSR_CAT_L3_CODE (DATA) => XC_PSR_CAT_L3_CBM_CODE (DATA)
* correct error message of passing -c and -d at the same time
Changes in v5:
* merge tools and docs patches
* replace EINVAL with ENXIO in libxl__psr_cat_log_err_msg
* revise options parsing in psr-cat-cbm-set and invalidate passing -c and -d at the same time
* refine CDP status output codes in psr_cat_hwinfo
* adjust CBM output format in command xl psr-cat-show
* docs revision
Example of new output format for command xl psr-cat-show:
*** CAT-only ***
Socket ID : 0
L3 Cache : 56320KB
Default CBM : 0xfffff
ID NAME CBM
0 Domain-0 0xfffff
1 centos.hvm 0xfffff*** CDP enabled ***
Socket ID : 0
L3 Cache : 56320KB
Default CBM : 0xfffff
ID NAME CBM (code) CBM (data)
0 Domain-0 0xfffff 0xfffff
1 centos.hvm 0xfffff 0xfffff
---
docs/man/xl.pod.1 | 15 +++++++++++
docs/misc/xl-psr.markdown | 53 +++++++++++++++++++++++++++++++++++++++
tools/libxc/include/xenctrl.h | 7 ++++--
tools/libxc/xc_psr.c | 17 ++++++++++++-
tools/libxl/libxl.h | 7 ++++++
tools/libxl/libxl_psr.c | 5 +++-
tools/libxl/libxl_types.idl | 3 +++
tools/libxl/xl_cmdimpl.c | 58 +++++++++++++++++++++++++++++++++++--------
tools/libxl/xl_cmdtable.c | 2 ++
9 files changed, 152 insertions(+), 15 deletions(-)docs/man/xl.pod.1 +15 -0
@@ -1518,6 +1518,13 @@ applications. In the Xen implementation, CAT is used to control cache allocation on VM basis. To enforce cache on a specific domain, just set capacity bitmasks (CBM) for the domain. +Intel Broadwell and later server platforms also offer Code/Data Prioritization +(CDP) for cache allocations, which support specifying code or data cache for +applications. CDP is used on a per VM basis in the Xen implementation. To +specify code or data CBM for the domain, CDP feature must be enabled and CBM +type options need to be specified when setting CBM, and the type options (code +and data) are mutually exclusive. + =over 4 =item B<psr-cat-cbm-set> [I<OPTIONS>] I<domain-id> I<cbm>
@@ -1533,6 +1540,14 @@ B<OPTIONS> Specify the socket to process, otherwise all sockets are processed. +=item B<-c>, B<--code> + +Set code CBM when CDP is enabled. + +=item B<-d>, B<--data> + +Set data CBM when CDP is enabled. + =back =item B<psr-cat-show> [I<domain-id>]
docs/misc/xl-psr.markdown +53 -0
59 lines not on this page · raw message
tools/libxc/include/xenctrl.h +5 -2
@@ -2827,7 +2827,9 @@ enum xc_psr_cmt_type {typedef enum xc_psr_cmt_type xc_psr_cmt_type ;enum xc_psr_cat_type { -XC_PSR_CAT_L3_CBM =1 , +XC_PSR_CAT_L3_CBM =1 , +XC_PSR_CAT_L3_CBM_CODE =2 , +XC_PSR_CAT_L3_CBM_DATA =3 , };typedef enum xc_psr_cat_type xc_psr_cat_type ;
@@ -2853,7 +2855,8 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid, xc_psr_cat_type type, uint32_t target, uint64_t *data);int xc_psr_cat_get_l3_info (xc_interface *xch,uint32_t socket, -uint32_t *cos_max,uint32_t *cbm_len); +uint32_t *cos_max,uint32_t *cbm_len, +bool *cdp_enabled);#endif #endif /* XENCTRL_H */
tools/libxc/xc_psr.c +16 -1
@@ -260,6 +260,12 @@ int xc_psr_cat_set_domain_data(xc_interface *xch, uint32_t domid,case XC_PSR_CAT_L3_CBM : cmd =XEN_DOMCTL_PSR_CAT_OP_SET_L3_CBM ;break ; +case XC_PSR_CAT_L3_CBM_CODE : + cmd =XEN_DOMCTL_PSR_CAT_OP_SET_L3_CODE ; +break ; +case XC_PSR_CAT_L3_CBM_DATA : + cmd =XEN_DOMCTL_PSR_CAT_OP_SET_L3_DATA ; +break ;default : errno =EINVAL ;return -1 ;
@@ -287,6 +293,12 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid,case XC_PSR_CAT_L3_CBM : cmd =XEN_DOMCTL_PSR_CAT_OP_GET_L3_CBM ;break ; +case XC_PSR_CAT_L3_CBM_CODE : + cmd =XEN_DOMCTL_PSR_CAT_OP_GET_L3_CODE ; +break ; +case XC_PSR_CAT_L3_CBM_DATA : + cmd =XEN_DOMCTL_PSR_CAT_OP_GET_L3_DATA ; +break ;default : errno =EINVAL ;return -1 ;
@@ -306,7 +318,8 @@ int xc_psr_cat_get_domain_data(xc_interface *xch, uint32_t domid, }int xc_psr_cat_get_l3_info (xc_interface *xch,uint32_t socket, -uint32_t *cos_max,uint32_t *cbm_len) +uint32_t *cos_max,uint32_t *cbm_len, +bool *cdp_enabled) {int rc;DECLARE_SYSCTL ;
@@ -320,6 +333,8 @@ int xc_psr_cat_get_l3_info(xc_interface *xch, uint32_t socket, { *cos_max = sysctl.u.psr_cat_op.u.l3_info.cos_max; *cbm_len = sysctl.u.psr_cat_op.u.l3_info.cbm_len; + *cdp_enabled = sysctl.u.psr_cat_op.u.l3_info.flags & +XEN_SYSCTL_PSR_CAT_L3_CDP ; }return rc;
tools/libxl/libxl.h +7 -0
@@ -803,6 +803,13 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, libxl_mac *src);* If this is defined, the Cache Allocation Technology feature is supported. */ #define LIBXL_HAVE_PSR_CAT 1 + +/* +* LIBXL_HAVE_PSR_CDP +* +* If this is defined, the Code and Data Prioritization feature is supported. +*/ +#define LIBXL_HAVE_PSR_CDP 1#endif /*
tools/libxl/libxl_psr.c +4 -1
@@ -87,6 +87,9 @@ static void libxl__psr_cat_log_err_msg(libxl__gc *gc, int err)case EEXIST : msg ="The same CBM is already set to this domain" ;break ; +case ENXIO : + msg ="Unable to set code or data CBM when CDP is disabled" ; +break ;default :libxl__psr_log_err_msg (gc, err);
@@ -363,7 +366,7 @@ int libxl_psr_cat_get_l3_info(libxl_ctx *ctx, libxl_psr_cat_info **info,libxl_for_each_set_bit (socketid, socketmap) { ptr[i].id = socketid;if (xc_psr_cat_get_l3_info (ctx->xch, socketid, &ptr[i].cos_max, - &ptr[i].cbm_len)) { + &ptr[i].cbm_len, &ptr[i].cdp_enabled)) {libxl__psr_cat_log_err_msg (gc, errno); rc =ERROR_FAIL ;free (ptr);
tools/libxl/libxl_types.idl +3 -0
@@ -797,10 +797,13 @@ libxl_psr_cmt_type = Enumeration("psr_cmt_type", [ libxl_psr_cbm_type = Enumeration("psr_cbm_type", [ (0, "UNKNOWN"), (1, "L3_CBM"), + (2, "L3_CODE"), + (3, "L3_DATA"), ]) libxl_psr_cat_info = Struct("psr_cat_info", [ ("id", uint32), ("cos_max", uint32), ("cbm_len", uint32), + ("cdp_enabled", bool), ])
tools/libxl/xl_cmdimpl.c +47 -11
124 lines not on this page · raw message
tools/libxl/xl_cmdtable.c +2 -0
@@ -543,6 +543,8 @@ struct cmd_spec cmd_table[] = {"Set cache capacity bitmasks(CBM) for a domain" ,"[options] <Domain> <CBM>" ,"-s <socket> Specify the socket to process, otherwise all sockets are processed\n" +"-c Set code CBM if CDP is supported\n" +"-d Set data CBM if CDP is supported\n" }, { "psr-cat-show", &main_psr_cat_show, 0, 1,
Signature · 1 line
1.9.1
Re: [PATCH v6 3/3] tools & docs: add tools and docs support for Intel CDP
He Chen writes ("[PATCH v6 3/3] tools & docs: add tools and docs support for Intel CDP"):
Quoted 6 lines
This is the xl/xc changes to support Intel Code/Data Prioritization. CAT xl commands to set/get CBMs are extended to support CDP. Add new CDP options with CAT commands in xl interface man page. Add description of CDP in xl-psr.markdown.
Signed-off-by He Chen <[email protected]>
I have reviewed this with my tools hat on. I am not an expert in this area. It's possible that the docs could be improved but overall I'm content.
I have one very minor nit. Several lines in the docs are too long, for example:
Quoted 1 line
+The xl interfaces are the same with that of CAT. The difference is that \
CBM type
Quoted 1 line
+can be passed as option to set code CBM or data CBM.
This leads to wrap damage when reviewing patches and editing docs. Can you rewrap them, preferably to 75 or fewer ?
I'm expecting the semantics and hypercall interfaces etc. to be reviewed by hypervisor maintainers.
Thanks,
Ian.
Re: [PATCH v6 3/3] tools & docs: add tools and docs support for Intel CDP
In reply to He ChenQuoted 11 lines · He Chen
This is the xl/xc changes to support Intel Code/Data Prioritization. CAT xl commands to set/get CBMs are extended to support CDP. Add new CDP options with CAT commands in xl interface man page. Add description of CDP in xl-psr.markdown.
Signed-off-by He Chen <[email protected]>---
Changes in v6:
* separate CBM headings in the output of xl psr-cat-show
* revert the numbers of SDM chapter in xl-psr.markdown
*XC_PSR_CAT_L3_CODE(DATA) =>XC_PSR_CAT_L3_CBM_CODE(DATA)
In this comment on v5 I mentioned that it was more important for libxl than libxc, I'm afraid that what I was trying to say was that the libxl names needed changing as well. Sorry for not clearly saying so.
Quoted 2 lines
+For more detailed information please refer to Intel SDM chapter +"17.15 - Platform Shared Resource Control: Cache Allocation Technology".
Looks like you missed deleting this section number.
Quoted 5 lines
tools/libxl/xl_cmdimpl.c +0
-1@@ -8454,20 +8457,30 @@ static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid)printf ("%5d%25s" , domid, domain_name);free (domain_name); - if (!libxl_psr_cat_get_cbm(ctx, domid, LIBXL_PSR_CBM_TYPE_L3_CBM,
tools/libxl/xl_cmdimpl.c static void psr_cat_print_one_domain_cbm(uint32_t domid, uint32_t socketid) · line 8467
Quoted 10 lines
tools/libxl/xl_cmdimpl.c +8
-2- socketid, &cbm)) -printf ("%#16" PRIx64, cbm); +if (!cdp_enabled) { +if (!libxl_psr_cat_get_cbm (ctx, domid,LIBXL_PSR_CBM_TYPE_L3_CBM , + socketid, &cbm)) +printf ("%#16" PRIx64, cbm); + }else { +if (!libxl_psr_cat_get_cbm (ctx, domid,LIBXL_PSR_CBM_TYPE_L3_CODE , + socketid, &cbm)) +printf ("%#16" PRIx64, cbm);
If this libxl_psr_cat_get_cbm(CODE) fails for some reason then this prints nothing, meaning that the following libxl_psr_cat_get_cbm(DATA) will actually print into the code column.
I think you should add an:
else
printf("%16s", "error")(or perhaps "%-16s" for alignment, please check which looks best and decide)
Please also refactor all three of these nearly identical:
if (!libxl_psr_cat_get_cbm(ctx, domid, LIBXL_PSR_CBM_TYPE_L3_???,
socketid, &cbm))
printf("%#16"PRIx64, cbm);blocks into a helper which takes the PSR_CBM_TYPE as an argument and prints either the result or the error string with the appropriate width.
Quoted 5 lines
tools/libxl/xl_cmdimpl.c +4
-0+if (!libxl_psr_cat_get_cbm (ctx, domid,LIBXL_PSR_CBM_TYPE_L3_DATA , + socketid, &cbm)) +printf ("%#16" PRIx64, cbm); + }