selftests/resctrl: Fixes and improvements focused on recent Intel platforms
Subsystems (2)
Reinette Chatre
maintainercover selftests/resctrl: Fixes and improvements focused on recent Intel platforms
This series is not intended to be merged yet since it depends on new perf sysfs files that have not yet been merged.
The resctrl selftests fail on recent Intel platforms. Intermittent failures in the CAT test and permanent failures of MBM and MBA tests on new platforms like Sierra Forest and Granite Rapids.
The MBM and MBA resctrl selftests both generate memory traffic and compare the memory bandwidth measurements between the iMC PMUs and MBM to determine pass or fail. Both these tests are failing on recent platforms like Sierra Forest and Granite Rapids that have two events that need to be read and combined for a total memory bandwidth count instead of the single event available on earlier platforms.
resctrl selftests prefer to obtain event details via sysfs instead of adding model specific details on which events to read. Enhancements to perf to expose the new event details are in progress:
https://lore.kernel.org/lkml/[email protected]/
This series demonstrates use of the new proposed sysfs interface to perf to obtain to obtain accurate iMC read memory bandwidth measurements.
An additional issue with all the tests is that these selftests are part performance tests and determine pass/fail on performance heuristics selected after running the tests on a variety of platforms. When new platforms arrive the previous heuristics may cause the tests to fail. These failures are not because of an issue with the resctrl subsystem the tests intend to test but because of the architectural changes in the new platforms.
Adapt the resctrl tests to not be as sensitive to architectural changes while adjusting the remaining heuristics to ensure tests pass on a variety of platforms. More details in individual patches.
Tested by running 100 iterations of all tests on Emerald Rapids, Granite Rapids, Sapphire Rapids, Ice Lake, Sierra Forest, and Broadwell.
This series is based on:
https://lore.kernel.org/lkml/[email protected]/
and
https://lore.kernel.org/lkml/084e82b5c29d75f16f24af8768d50d39ba0118a5.1769101788.git.reinette.chatre@intel.com/
applied to
next branch of git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git
Reinette Chatre (8):
selftests/resctrl: Do not store iMC counter value in counter config
structure
selftests/resctrl: Prepare for parsing multiple events per iMC selftests/resctrl: Support multiple events associated with iMC selftests/resctrl: Increase size of buffer used in MBM and MBA tests selftests/resctrl: Raise threshold at which MBM and PMU values are
compared
selftests/resctrl: Remove requirement on cache miss rate selftests/resctrl: Simplify perf usage in CAT test selftests/resctrl: Reduce L2 impact on CAT test
tools/testing/selftests/resctrl/cache.c | 17 +-- tools/testing/selftests/resctrl/cat_test.c | 43 ++---- tools/testing/selftests/resctrl/fill_buf.c | 4 +- tools/testing/selftests/resctrl/mba_test.c | 2 +- tools/testing/selftests/resctrl/mbm_test.c | 2 +- tools/testing/selftests/resctrl/resctrl.h | 13 +- tools/testing/selftests/resctrl/resctrl_val.c | 129 +++++++++++++----- 7 files changed, 121 insertions(+), 89 deletions(-)
Signature · 1 line
2.50.1
Reinette Chatre
maintainerPATCH 1/8 selftests/resctrl: Do not store iMC counter value in counter config structure
The MBM and MBA tests compare MBM memory bandwidth measurements against the memory bandwidth event values obtained from each memory controller's PMU. The memory bandwidth event settings are discovered from the memory controller details found in /sys/bus/event_source/devices/uncore_imc_N and stored in struct imc_counter_config.
In addition to event settings struct imc_counter_config contains imc_counter_config::return_value in which the associated event value is stored on every read.
The event value is consumed and immediately recorded at regular intervals. The stored value is never consumed afterwards, making its storage as part of event configuration unnecessary.
Remove the return_value member from struct imc_counter_config. Instead just use a local variable for use during event reading.
--- tools/testing/selftests/resctrl/resctrl_val.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
tools/testing/selftests/resctrl/resctrl_val.c +5 -6
@@ -32,7 +32,6 @@ struct imc_counter_config {__u64 event;__u64 umask;struct perf_event_attr pe; -struct membw_read_format return_value;int fd; };
@@ -312,23 +311,23 @@ static int get_read_mem_bw_imc(float *bw_imc)* Take overflow into consideration before calculating total bandwidth. */ for (imc =0 ; imc < imcs; imc++) { +struct membw_read_format return_value;struct imc_counter_config *r = &imc_counters_config[imc]; -if (read (r->fd, &r->return_value, -sizeof (struct membw_read_format )) ==-1 ) { +if (read (r->fd, &return_value,sizeof (return_value)) ==-1 ) {ksft_perror ("Couldn't get read bandwidth through iMC" );return -1 ; } -__u64 r_time_enabled = r->return_value.time_enabled; -__u64 r_time_running = r->return_value.time_running; +__u64 r_time_enabled = return_value.time_enabled; +__u64 r_time_running = return_value.time_running;if (r_time_enabled != r_time_running) of_mul_read = (float )r_time_enabled / (float )r_time_running; - reads += r->return_value.value * of_mul_read *SCALE ; + reads += return_value.value * of_mul_read *SCALE ; } *bw_imc = reads;
Signature · 1 line
2.50.1
Reinette Chatre
maintainerPATCH 2/8 selftests/resctrl: Prepare for parsing multiple events per iMC
In reply to Reinette ChatreThe events needed to read memory bandwidth are discovered by iterating over every memory controller (iMC) within /sys/bus/event_source/devices. Each iMC's PMU is assumed to have one event to measure read memory bandwidth that is represented by the sysfs cas_count_read file. The event's configuration is read from "cas_count_read" and stored as an element of imc_counters_config[] by read_from_imc_dir() that receives the index of the array where to store the configuration as argument.
It is possible that an iMC's PMU may have more than one event that should be used to measure memory bandwidth.
Change semantics to not provide the index of the array to read_from_imc_dir() but instead a pointer to the index. This enables read_from_imc_dir() to store configurations for more than one event by incrementing the index to imc_counters_config[] itself.
Ensure that the same type is consistently used for the index as it is passed around during counter configuration.
--- tools/testing/selftests/resctrl/resctrl_val.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
tools/testing/selftests/resctrl/resctrl_val.c +6 -6
@@ -73,7 +73,7 @@ static void read_mem_bw_ioctl_perf_event_ioc_disable(int i)* @cas_count_cfg: Config * @count: iMC number */ -static void get_read_event_and_umask (char *cas_count_cfg,int count) +static void get_read_event_and_umask (char *cas_count_cfg,unsigned int count) {char *token[MAX_TOKENS ];int i =0 ;
@@ -110,7 +110,7 @@ static int open_perf_read_event(int i, int cpu_no) }/* Get type and config of an iMC counter's read event. */ -static int read_from_imc_dir (char *imc_dir,int count) +static int read_from_imc_dir (char *imc_dir,unsigned int *count) {char cas_count_cfg[1024 ], imc_counter_cfg[1024 ], imc_counter_type[1024 ];FILE *fp;
@@ -123,7 +123,7 @@ static int read_from_imc_dir(char *imc_dir, int count)return -1 ; } -if (fscanf (fp,"%u" , &imc_counters_config[count].type) <=0 ) { +if (fscanf (fp,"%u" , &imc_counters_config[*count].type) <=0 ) {ksft_perror ("Could not get iMC type" );fclose (fp);
@@ -147,7 +147,8 @@ static int read_from_imc_dir(char *imc_dir, int count) }fclose (fp); -get_read_event_and_umask (cas_count_cfg, count); +get_read_event_and_umask (cas_count_cfg, *count); + *count +=1 ;return 0 ; }
@@ -196,13 +197,12 @@ static int num_of_imcs(void)if (temp[0 ] >='0' && temp[0 ] <='9' ) {sprintf (imc_dir,"%s/%s/" ,DYN_PMU_PATH , ep->d_name); - ret =read_from_imc_dir (imc_dir, count); + ret =read_from_imc_dir (imc_dir, &count);if (ret) {closedir (dp);return ret; } - count++; } }closedir (dp);
Signature · 1 line
2.50.1
Reinette Chatre
maintainerPATCH 3/8 selftests/resctrl: Support multiple events associated with iMC
In reply to Reinette ChatreThe resctrl selftests discover needed parameters to perf_event_open() via sysfs. The PMU associated with every memory controller (iMC) is discovered via the /sys/bus/event_source/devices/uncore_imc_N/type file while the read memory bandwidth event type and umask is discovered via /sys/bus/event_source/devices/uncore_imc_N/events/cas_count_read.
Newer systems may have multiple events that expose read memory bandwidth. For example,
/sys/bus/event_source/devices/uncore_imc_N/events/cas_count_read_sch0 /sys/bus/event_source/devices/uncore_imc_N/events/cas_count_read_sch1
Support parsing of iMC PMU properties when the PMU may have multiple events to measure read memory bandwidth. The PMU only needs to be discovered once. Split the parsing of event details from actual PMU discovery in order to loop over all events associated with the PMU. Match all events with the cas_count_read prefix instead of requiring there to be one file with that name.
Make the parsing code more robust. With strings passed around to create needed paths, use snprintf() instead of sprintf() to ensure there is always enough space to create the path. Ensure there is enough room in imc_counters_config[] before attempting to add an entry.
--- tools/testing/selftests/resctrl/resctrl_val.c | 112 ++++++++++++++---- 1 file changed, 90 insertions(+), 22 deletions(-)
tools/testing/selftests/resctrl/resctrl_val.c +90 -22
@@ -11,10 +11,10 @@#include "resctrl.h" #define UNCORE_IMC "uncore_imc" -#define READ_FILE_NAME "events/cas_count_read" +#define READ_FILE_NAME "cas_count_read"#define DYN_PMU_PATH "/sys/bus/event_source/devices"#define SCALE 0.00006103515625 -#define MAX_IMCS 20 +#define MAX_IMCS 40#define MAX_TOKENS 5#define CON_MBM_LOCAL_BYTES_PATH \
@@ -109,21 +109,102 @@ static int open_perf_read_event(int i, int cpu_no)return 0 ; } +static int parse_imc_read_bw_events (char *imc_dir,unsigned int type, +unsigned int *count) +{ +char imc_events[1024 ], imc_counter_cfg[1024 ], cas_count_cfg[1024 ]; +unsigned int org_count = *count; +struct dirent *ep; +int path_len; +int ret =-1 ; +FILE *fp; +DIR *dp; + + path_len =snprintf (imc_events,sizeof (imc_events),"%sevents" , imc_dir); +if (path_len >=sizeof (imc_events)) { +ksft_print_msg ("Unable to create path to %sevents\n" , imc_dir); +return -1 ; + } + dp =opendir (imc_events); +if (dp) { +while ((ep =readdir (dp))) { +/* +* Parse all event files with READ_FILE_NAME +* prefix that contain the event number and umask. +* Skip files containing "." that contain unused +* properties of event. +*/ +if (!strstr (ep->d_name,READ_FILE_NAME ) || +strchr (ep->d_name,'.' )) +continue ; + + path_len =snprintf (imc_counter_cfg,sizeof (imc_counter_cfg), +"%s/%s" , imc_events, ep->d_name); +if (path_len >=sizeof (imc_counter_cfg)) { +ksft_print_msg ("Unable to create path to %s/%s\n" , + imc_events, ep->d_name); + goto out_close; + } + fp =fopen (imc_counter_cfg,"r" ); +if (!fp) { +ksft_perror ("Failed to open iMC config file" ); + goto out_close; + } +if (fscanf (fp,"%1023s" , cas_count_cfg) <=0 ) { +ksft_perror ("Could not get iMC cas count read" ); +fclose (fp); + goto out_close; + } +fclose (fp); +if (*count >=MAX_IMCS ) { +ksft_print_msg ("Maximum iMC count exceeded\n" ); + goto out_close; + } + + imc_counters_config[*count].type = type; +get_read_event_and_umask (cas_count_cfg, *count); +/* Do not fail after incrementing *count. */ + *count +=1 ; + } +if (*count == org_count) { +ksft_print_msg ("Unable to find events in %s\n" , imc_events); + goto out_close; + } + }else { +ksft_perror ("Unable to open PMU events directory" ); + goto out; + } + ret =0 ; +out_close: +closedir (dp); +out: +return ret; +} +/* Get type and config of an iMC counter's read event. */ static int read_from_imc_dir (char *imc_dir,unsigned int *count) { -char cas_count_cfg[1024 ], imc_counter_cfg[1024 ], imc_counter_type[1024 ]; +char imc_counter_type[1024 ]; +unsigned int type; +int path_len;FILE *fp; +int ret;/* Get type of iMC counter */ -sprintf (imc_counter_type,"%s%s" , imc_dir,"type" ); + path_len =snprintf (imc_counter_type,sizeof (imc_counter_type), +"%s%s" , imc_dir,"type" ); +if (path_len >=sizeof (imc_counter_type)) { +ksft_print_msg ("Unable to create path to %s%s\n" , + imc_dir,"type" ); +return -1 ; + } fp =fopen (imc_counter_type,"r" );if (!fp) {ksft_perror ("Failed to open iMC counter type file" );return -1 ; } -if (fscanf (fp,"%u" , &imc_counters_config[*count].type) <=0 ) { +if (fscanf (fp,"%u" , &type) <=0 ) {ksft_perror ("Could not get iMC type" );fclose (fp);
@@ -131,24 +212,11 @@ static int read_from_imc_dir(char *imc_dir, unsigned int *count) }fclose (fp); -/* Get read config */ -sprintf (imc_counter_cfg,"%s%s" , imc_dir,READ_FILE_NAME ); - fp =fopen (imc_counter_cfg,"r" ); -if (!fp) { -ksft_perror ("Failed to open iMC config file" ); - -return -1 ; - } -if (fscanf (fp,"%1023s" , cas_count_cfg) <=0 ) { -ksft_perror ("Could not get iMC cas count read" ); -fclose (fp); - -return -1 ; + ret =parse_imc_read_bw_events (imc_dir, type, count); +if (ret) { +ksft_print_msg ("Unable to parse bandwidth event and umask\n" ); +return ret; } -fclose (fp); - -get_read_event_and_umask (cas_count_cfg, *count); - *count +=1 ;return 0 ; }
Signature · 1 line
2.50.1
Reinette Chatre
maintainerPATCH 4/8 selftests/resctrl: Increase size of buffer used in MBM and MBA tests
In reply to Reinette ChatreErrata for Sierra Forest [1] (SRF42) and Granite Rapids[2] (GNR12) describe the problem that MBM on Intel RDT may overcount memory bandwidth measurements. The resctrl tests compare memory bandwidth reported by iMC PMU to that reported by MBM causing the tests to fail on these systems depending on the settings of the platform related to the errata.
Since the resctrl tests need to run under various conditions it is not possible to ensure system settings are such that MBM will not overcount. It has been observed that the overcounting can be controlled via the buffer size used in the MBM and MBA tests that rely on comparisons between iMC PMU and MBM measurements.
Running the MBM test on affected platforms with different buffer sizes it can be observed that the difference between iMC PMU and MBM counts reduce as the buffer size increases. After increasing the buffer size to more than 4X the differences between iMC PMU and MBM become insignificant.
Increase the buffer size used in MBM and MBA tests to 4X L3 size to reduce possibility of tests failing due to difference in counts reported by iMC PMU and MBM.
--- tools/testing/selftests/resctrl/fill_buf.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/resctrl/fill_buf.c +2 -2
@@ -139,6 +139,6 @@ ssize_t get_fill_buf_size(int cpu_no, const char *cache_type)if (ret)return ret; -return cache_total_size *2 >MINIMUM_SPAN ? - cache_total_size *2 :MINIMUM_SPAN ; +return cache_total_size *4 >MINIMUM_SPAN ? + cache_total_size *4 :MINIMUM_SPAN ; }
Signature · 1 line
2.50.1
Reinette Chatre
maintainerPATCH 5/8 selftests/resctrl: Raise threshold at which MBM and PMU values are compared
In reply to Reinette Chatrecommit 501cfdba0a40 ("selftests/resctrl: Do not compare performance counters and resctrl at low bandwidth") introduced a threshold under which memory bandwidth values from MBM and performance counters are not compared. This is needed because MBM and the PMUs do not have an identical view of memory bandwidth since PMUs can count all memory traffic while MBM does not count "overhead" (for example RAS) traffic that cannot be attributed to an RMID. As a ratio this difference in view of memory bandwidth is pronounced at low memory bandwidths.
The 750MiB threshold was chosen arbitrarily after comparisons on different platforms. Exposed to more platforms after introduction this threshold has proven to be inadequate.
Having accurate comparison between performance counters and MBM requires careful management of system load as well as control of features that introduce extra memory traffic, for example, patrol scrub. This is not appropriate for the resctrl selftests that are intended to run on a variety of systems with various configurations.
Increase the memory bandwidth threshold under which no comparison is made between performance counters and MBM. Add additional leniency by increasing the percentage of difference that will be tolerated between these counts.
There is no impact to the validity of the resctrl selftests results as a measure of resctrl subsystem health.
--- tools/testing/selftests/resctrl/mba_test.c | 2 +- tools/testing/selftests/resctrl/mbm_test.c | 2 +- tools/testing/selftests/resctrl/resctrl.h | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
tools/testing/selftests/resctrl/mba_test.c +1 -1
@@ -12,7 +12,7 @@#define RESULT_FILE_NAME "result_mba"#define NUM_OF_RUNS 5 -#define MAX_DIFF_PERCENT 8 +#define MAX_DIFF_PERCENT 15#define ALLOCATION_MAX 100#define ALLOCATION_MIN 10#define ALLOCATION_STEP 10
tools/testing/selftests/resctrl/mbm_test.c +1 -1
@@ -11,7 +11,7 @@#include "resctrl.h" #define RESULT_FILE_NAME "result_mbm" -#define MAX_DIFF_PERCENT 8 +#define MAX_DIFF_PERCENT 15#define NUM_OF_RUNS 5 static int
tools/testing/selftests/resctrl/resctrl.h +1 -1
@@ -55,7 +55,7 @@* and MBM respectively, for instance generating "overhead" traffic which * is not counted against any specific RMID. */ -#define THROTTLE_THRESHOLD 750 +#define THROTTLE_THRESHOLD 2500/* * fill_buf_param: "fill_buf" benchmark parameters
Signature · 1 line
2.50.1
Reinette Chatre
maintainerPATCH 6/8 selftests/resctrl: Remove requirement on cache miss rate
In reply to Reinette ChatreAs the CAT test reads the same buffer into different sized cache portions it compares the number of cache misses against an expected percentage based on the size of the cache portion.
Systems and test conditions vary. The CAT test is a test of resctrl subsystem health and not a test of the hardware architecture so it is not required to place requirements on the size of the difference in cache misses, just that the number of cache misses when reading a buffer increase as the cache portion used for the buffer decreases.
Remove additional constraint on how big the difference between cache misses should be as the cache portion size changes. Only test that the cache misses increase as the cache portion size decreases. This remains a good sanity check of resctrl subsystem health while reducing impact of hardware architectural differences and the various conditions under which the test may run.
Increase the size difference between cache portions to additionally avoid any consequences resulting from smaller increments.
--- tools/testing/selftests/resctrl/cat_test.c | 33 ++++------------------ 1 file changed, 5 insertions(+), 28 deletions(-)
tools/testing/selftests/resctrl/cat_test.c +5 -28
@@ -14,42 +14,20 @@#define RESULT_FILE_NAME "result_cat"#define NUM_OF_RUNS 5 -/* -* Minimum difference in LLC misses between a test with n+1 bits CBM to the -* test with n bits is MIN_DIFF_PERCENT_PER_BIT * (n - 1). With e.g. 5 vs 4 -* bits in the CBM mask, the minimum difference must be at least -* MIN_DIFF_PERCENT_PER_BIT * (4 - 1) = 3 percent. -* -* The relationship between number of used CBM bits and difference in LLC -* misses is not expected to be linear. With a small number of bits, the -* margin is smaller than with larger number of bits. For selftest purposes, -* however, linear approach is enough because ultimately only pass/fail -* decision has to be made and distinction between strong and stronger -* signal is irrelevant. -*/ -#define MIN_DIFF_PERCENT_PER_BIT 1UL -static int show_results_info (__u64 sum_llc_val,int no_of_bits,unsigned long cache_span, -unsigned long min_diff_percent,unsigned long num_of_runs,bool platform,__s64 *prev_avg_llc_val) {__u64 avg_llc_val =0 ; -float avg_diff;int ret =0 ; avg_llc_val = sum_llc_val / num_of_runs;if (*prev_avg_llc_val) { -float delta = (__s64)(avg_llc_val - *prev_avg_llc_val); - - avg_diff = delta / *prev_avg_llc_val; - ret = platform && (avg_diff *100 ) < (float )min_diff_percent; - -ksft_print_msg ("%s Check cache miss rate changed more than %.1f%%\n" , - ret ?"Fail:" :"Pass:" , (float )min_diff_percent); + ret = platform && (avg_llc_val < *prev_avg_llc_val); -ksft_print_msg ("Percent diff=%.1f\n" , avg_diff *100 ); +ksft_print_msg ("%s Check cache miss rate increased\n" , + ret ?"Fail:" :"Pass:" ); } *prev_avg_llc_val = avg_llc_val;
@@ -58,10 +36,10 @@ static int show_results_info(__u64 sum_llc_val, int no_of_bits,return ret; } -/* Remove the highest bit from CBM */ +/* Remove the highest bits from CBM */ static unsigned long next_mask (unsigned long current_mask) { -return current_mask & (current_mask >>1 ); +return current_mask & (current_mask >>2 ); } static int check_results(struct resctrl_val_param *param, const char *cache_type,
@@ -112,7 +90,6 @@ static int check_results(struct resctrl_val_param *param, const char *cache_type ret =show_results_info (sum_llc_perf_miss, bits, alloc_size /64 , -MIN_DIFF_PERCENT_PER_BIT * (bits -1 ), runs,get_vendor () ==ARCH_INTEL , &prev_avg_llc_val);if (ret)
Signature · 1 line
2.50.1
Reinette Chatre
maintainerPATCH 7/8 selftests/resctrl: Simplify perf usage in CAT test
In reply to Reinette ChatreThe CAT test relies on the PERF_COUNT_HW_CACHE_MISSES event to determine if modifying a cache portion size is successful. This event is configured to report the data as part of an event group, but no other events are added to the group.
Remove the unnecessary PERF_FORMAT_GROUP format setting. This eliminates the need for struct perf_event_read and results in read() of the associated file descriptor to return just one value associated with the PERF_COUNT_HW_CACHE_MISSES event of interest.
--- tools/testing/selftests/resctrl/cache.c | 17 +++++------------ tools/testing/selftests/resctrl/cat_test.c | 4 +--- tools/testing/selftests/resctrl/resctrl.h | 11 +---------- 3 files changed, 7 insertions(+), 25 deletions(-)
tools/testing/selftests/resctrl/cache.c +5 -12
@@ -10,7 +10,6 @@ void perf_event_attr_initialize(struct perf_event_attr *pea, __u64 config)memset (pea,0 ,sizeof (*pea)); pea->type =PERF_TYPE_HARDWARE ; pea->size =sizeof (*pea); - pea->read_format =PERF_FORMAT_GROUP ; pea->exclude_kernel =1 ; pea->exclude_hv =1 ; pea->exclude_idle =1 ;
@@ -37,19 +36,13 @@ int perf_event_reset_enable(int pe_fd)return 0 ; } -void perf_event_initialize_read_format (struct perf_event_read *pe_read) -{ -memset (pe_read,0 ,sizeof (*pe_read)); - pe_read->nr =1 ; -} -int perf_open (struct perf_event_attr *pea,pid_t pid,int cpu_no) {int pe_fd; pe_fd =perf_event_open (pea, pid, cpu_no,-1 ,PERF_FLAG_FD_CLOEXEC );if (pe_fd ==-1 ) { -ksft_perror ("Error opening leader" ); +ksft_perror ("Unable to set up performance monitoring" );return -1 ; }
@@ -132,9 +125,9 @@ static int print_results_cache(const char *filename, pid_t bm_pid, __u64 llc_val* * Return: =0 on success. <0 on failure. */ -int perf_event_measure (int pe_fd,struct perf_event_read *pe_read, -const char *filename,pid_t bm_pid) +int perf_event_measure (int pe_fd,const char *filename,pid_t bm_pid) { +__u64 value;int ret;/* Stop counters after one span to get miss rate */
@@ -142,13 +135,13 @@ int perf_event_measure(int pe_fd, struct perf_event_read *pe_read,if (ret <0 )return ret; - ret =read (pe_fd, pe_read,sizeof (*pe_read)); + ret =read (pe_fd, &value,sizeof (value));if (ret ==-1 ) {ksft_perror ("Could not get perf value" );return -1 ; } -return print_results_cache (filename, bm_pid, pe_read->values[0 ].value); +return print_results_cache (filename, bm_pid, value); }/*
tools/testing/selftests/resctrl/cat_test.c +1 -3
@@ -135,7 +135,6 @@ static int cat_test(const struct resctrl_test *test,struct resctrl_val_param *param, size_t span, unsigned long current_mask) { - struct perf_event_read pe_read; struct perf_event_attr pea;cpu_set_t old_affinity;unsigned char *buf;
@@ -159,7 +158,6 @@ static int cat_test(const struct resctrl_test *test, goto reset_affinity;perf_event_attr_initialize (&pea,PERF_COUNT_HW_CACHE_MISSES ); -perf_event_initialize_read_format (&pe_read); pe_fd =perf_open (&pea, bm_pid, uparams->cpu);if (pe_fd <0 ) { ret =-1 ;
@@ -192,7 +190,7 @@ static int cat_test(const struct resctrl_test *test,fill_cache_read (buf, span, true); - ret =perf_event_measure (pe_fd, &pe_read, param->filename, bm_pid); + ret =perf_event_measure (pe_fd, param->filename, bm_pid);if (ret) goto free_buf; }
tools/testing/selftests/resctrl/resctrl.h +1 -10
@@ -148,13 +148,6 @@ struct resctrl_val_param {struct fill_buf_param *fill_buf; }; -struct perf_event_read { -__u64 nr;/* The number of events */ -struct { -__u64 value;/* The value of the event */ - } values[2 ]; -}; -/* * Memory location that consumes values compiler must not optimize away. * Volatile ensures writes to this location cannot be optimized away by
@@ -210,11 +203,9 @@ unsigned int count_bits(unsigned long n);int snc_kernel_support (void );void perf_event_attr_initialize (struct perf_event_attr *pea,__u64 config); -void perf_event_initialize_read_format (struct perf_event_read *pe_read);int perf_open (struct perf_event_attr *pea,pid_t pid,int cpu_no);int perf_event_reset_enable (int pe_fd); -int perf_event_measure (int pe_fd,struct perf_event_read *pe_read, -const char *filename,pid_t bm_pid); +int perf_event_measure (int pe_fd,const char *filename,pid_t bm_pid);int measure_llc_resctrl (const char *filename,pid_t bm_pid);void show_cache_info (int no_of_bits,__u64 avg_llc_val,size_t cache_span,bool lines);
Signature · 1 line
2.50.1
Reinette Chatre
maintainerPATCH 8/8 selftests/resctrl: Reduce L2 impact on CAT test
In reply to Reinette ChatreThe L3 CAT test loads a buffer into cache that is proportional to the L3 size allocated for the workload and measures cache misses when accessing the buffer as a test of L3 occupancy. When loading the buffer it can be assumed that a portion of the buffer will be loaded into the L2 cache and depending on cache design may not be present in L3. It is thus possible for data to not be in L3 but also not trigger an L3 cache miss when accessed.
Reduce impact of L2 on the L3 CAT test by, if L2 allocation is supported, minimizing the portion of L2 that the workload can allocate into. This encourages most of buffer to be loaded into L3 and support better comparison between buffer size, cache portion, and cache misses when accessing the buffer.
--- tools/testing/selftests/resctrl/cat_test.c | 6 ++++++ 1 file changed, 6 insertions(+)
tools/testing/selftests/resctrl/cat_test.c +6 -0
@@ -157,6 +157,12 @@ static int cat_test(const struct resctrl_test *test,if (ret) goto reset_affinity; +if (!strcmp (test->resource,"L3" ) &&resctrl_resource_exists ("L2" )) { + ret =write_schemata (param->ctrlgrp,"0x1" , uparams->cpu,"L2" ); +if (ret) + goto reset_affinity; + } +perf_event_attr_initialize (&pea,PERF_COUNT_HW_CACHE_MISSES ); pe_fd =perf_open (&pea, bm_pid, uparams->cpu);if (pe_fd <0 ) {
Signature · 1 line
2.50.1
Re: [PATCH 2/8] selftests/resctrl: Prepare for parsing multiple events per iMC
In reply to Reinette ChatreQuoted 16 lines · Reinette Chatre
The events needed to read memory bandwidth are discovered by iterating over every memory controller (iMC) within
/sys/bus/event_source/devices. Each iMC's PMU is assumed to have one event to measure read memory bandwidth that is represented by the sysfscas_count_readfile. The event's configuration is read from "cas_count_read" and stored as an element of imc_counters_config[] byread_from_imc_dir()that receives the index of the array where to store the configuration as argument.It is possible that an iMC's PMU may have more than one event that should be used to measure memory bandwidth.
Change semantics to not provide the index of the array to
read_from_imc_dir()but instead a pointer to the index. This enablesread_from_imc_dir()to store configurations for more than one event by incrementing the index to imc_counters_config[] itself.
Quoted 4 lines
Ensure that the same type is consistently used for the index as it is passed around during counter configuration.
Signed-off-by Reinette Chatre <[email protected]>
Quoted 60 lines
--- tools/testing/selftests/resctrl/resctrl_val.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)tools/testing/selftests/resctrl/resctrl_val.c +6
-6@@ -73,7 +73,7 @@ static void read_mem_bw_ioctl_perf_event_ioc_disable(int i)* @cas_count_cfg: Config * @count: iMC number */ -static void get_read_event_and_umask (char *cas_count_cfg,int count) +static void get_read_event_and_umask (char *cas_count_cfg,unsigned int count) {char *token[MAX_TOKENS ];int i =0 ;@@ -110,7 +110,7 @@ static int open_perf_read_event(int i, int cpu_no) }/* Get type and config of an iMC counter's read event. */ -static int read_from_imc_dir (char *imc_dir,int count) +static int read_from_imc_dir (char *imc_dir,unsigned int *count) {char cas_count_cfg[1024 ], imc_counter_cfg[1024 ], imc_counter_type[1024 ];FILE *fp;@@ -123,7 +123,7 @@ static int read_from_imc_dir(char *imc_dir, int count)return -1 ; } -if (fscanf (fp,"%u" , &imc_counters_config[count].type) <=0 ) { +if (fscanf (fp,"%u" , &imc_counters_config[*count].type) <=0 ) {ksft_perror ("Could not get iMC type" );fclose (fp);@@ -147,7 +147,8 @@ static int read_from_imc_dir(char *imc_dir, int count) }fclose (fp); -get_read_event_and_umask (cas_count_cfg, count); +get_read_event_and_umask (cas_count_cfg, *count); + *count +=1 ;return 0 ; }@@ -196,13 +197,12 @@ static int num_of_imcs(void)if (temp[0 ] >='0' && temp[0 ] <='9' ) {sprintf (imc_dir,"%s/%s/" ,DYN_PMU_PATH , ep->d_name); - ret =read_from_imc_dir (imc_dir, count); + ret =read_from_imc_dir (imc_dir, &count);if (ret) {closedir (dp);return ret; } - count++; } }closedir (dp);
Re: [PATCH 3/8] selftests/resctrl: Support multiple events associated with iMC
In reply to Reinette ChatreQuoted 18 lines · Reinette Chatre
The resctrl selftests discover needed parameters to
perf_event_open()via sysfs. The PMU associated with every memory controller (iMC) is discovered via the/sys/bus/event_source/devices/uncore_imc_N/typefile while the read memory bandwidth event type and umask is discovered via/sys/bus/event_source/devices/uncore_imc_N/events/cas_count_read.Newer systems may have multiple events that expose read memory bandwidth. For example,
/sys/bus/event_source/devices/uncore_imc_N/events/cas_count_read_sch0/sys/bus/event_source/devices/uncore_imc_N/events/cas_count_read_sch1Support parsing of iMC PMU properties when the PMU may have multiple events to measure read memory bandwidth. The PMU only needs to be discovered once. Split the parsing of event details from actual PMU discovery in order to loop over all events associated with the PMU. Match all events with the
cas_count_readprefix instead of requiring there to be one file with that name.
Quoted 6 lines
Make the parsing code more robust. With strings passed around to create needed paths, use
snprintf()instead ofsprintf()to ensure there is always enough space to create the path. Ensure there is enough room in imc_counters_config[] before attempting to add an entry.Signed-off-by Reinette Chatre <[email protected]>
Quoted 156 lines
--- tools/testing/selftests/resctrl/resctrl_val.c | 112 ++++++++++++++---- 1 file changed, 90 insertions(+), 22 deletions(-)tools/testing/selftests/resctrl/resctrl_val.c +90
-22@@ -11,10 +11,10 @@#include "resctrl.h" #define UNCORE_IMC "uncore_imc" -#define READ_FILE_NAME "events/cas_count_read" +#define READ_FILE_NAME "cas_count_read"#define DYN_PMU_PATH "/sys/bus/event_source/devices"#define SCALE 0.00006103515625 -#define MAX_IMCS 20 +#define MAX_IMCS 40#define MAX_TOKENS 5#define CON_MBM_LOCAL_BYTES_PATH \@@ -109,21 +109,102 @@ static int open_perf_read_event(int i, int cpu_no)return 0 ; } +static int parse_imc_read_bw_events (char *imc_dir,unsigned int type, +unsigned int *count) +{ +char imc_events[1024 ], imc_counter_cfg[1024 ], cas_count_cfg[1024 ]; +unsigned int org_count = *count; +struct dirent *ep; +int path_len; +int ret =-1 ; +FILE *fp; +DIR *dp; + + path_len =snprintf (imc_events,sizeof (imc_events),"%sevents" , imc_dir); +if (path_len >=sizeof (imc_events)) { +ksft_print_msg ("Unable to create path to %sevents\n" , imc_dir); +return -1 ; + } + dp =opendir (imc_events); +if (dp) { +while ((ep =readdir (dp))) { +/* +* Parse all event files with READ_FILE_NAME +* prefix that contain the event number and umask. +* Skip files containing "." that contain unused +* properties of event. +*/ +if (!strstr (ep->d_name,READ_FILE_NAME ) || +strchr (ep->d_name,'.' )) +continue ; + + path_len =snprintf (imc_counter_cfg,sizeof (imc_counter_cfg), +"%s/%s" , imc_events, ep->d_name); +if (path_len >=sizeof (imc_counter_cfg)) { +ksft_print_msg ("Unable to create path to %s/%s\n" , + imc_events, ep->d_name); + goto out_close; + } + fp =fopen (imc_counter_cfg,"r" ); +if (!fp) { +ksft_perror ("Failed to open iMC config file" ); + goto out_close; + } +if (fscanf (fp,"%1023s" , cas_count_cfg) <=0 ) { +ksft_perror ("Could not get iMC cas count read" ); +fclose (fp); + goto out_close; + } +fclose (fp); +if (*count >=MAX_IMCS ) { +ksft_print_msg ("Maximum iMC count exceeded\n" ); + goto out_close; + } + + imc_counters_config[*count].type = type; +get_read_event_and_umask (cas_count_cfg, *count); +/* Do not fail after incrementing *count. */ + *count +=1 ; + } +if (*count == org_count) { +ksft_print_msg ("Unable to find events in %s\n" , imc_events); + goto out_close; + } + }else { +ksft_perror ("Unable to open PMU events directory" ); + goto out; + } + ret =0 ; +out_close: +closedir (dp); +out: +return ret; +} +/* Get type and config of an iMC counter's read event. */ static int read_from_imc_dir (char *imc_dir,unsigned int *count) { -char cas_count_cfg[1024 ], imc_counter_cfg[1024 ], imc_counter_type[1024 ]; +char imc_counter_type[1024 ]; +unsigned int type; +int path_len;FILE *fp; +int ret;/* Get type of iMC counter */ -sprintf (imc_counter_type,"%s%s" , imc_dir,"type" ); + path_len =snprintf (imc_counter_type,sizeof (imc_counter_type), +"%s%s" , imc_dir,"type" ); +if (path_len >=sizeof (imc_counter_type)) { +ksft_print_msg ("Unable to create path to %s%s\n" , + imc_dir,"type" ); +return -1 ; + } fp =fopen (imc_counter_type,"r" );if (!fp) {ksft_perror ("Failed to open iMC counter type file" );return -1 ; } -if (fscanf (fp,"%u" , &imc_counters_config[*count].type) <=0 ) { +if (fscanf (fp,"%u" , &type) <=0 ) {ksft_perror ("Could not get iMC type" );fclose (fp);@@ -131,24 +212,11 @@ static int read_from_imc_dir(char *imc_dir, unsigned int *count) }fclose (fp); -/* Get read config */ -sprintf (imc_counter_cfg,"%s%s" , imc_dir,READ_FILE_NAME ); - fp =fopen (imc_counter_cfg,"r" ); -if (!fp) { -ksft_perror ("Failed to open iMC config file" ); - -return -1 ; - } -if (fscanf (fp,"%1023s" , cas_count_cfg) <=0 ) { -ksft_perror ("Could not get iMC cas count read" ); -fclose (fp); - -return -1 ; + ret =parse_imc_read_bw_events (imc_dir, type, count); +if (ret) { +ksft_print_msg ("Unable to parse bandwidth event and umask\n" ); +return ret; } -fclose (fp); - -get_read_event_and_umask (cas_count_cfg, *count); - *count +=1 ;return 0 ; }
Reinette Chatre
maintainerRe: [PATCH 2/8] selftests/resctrl: Prepare for parsing multiple events per iMC
In reply to Chen, ZideHi Zide,
Quoted 17 lines · Zide
Quoted 16 lines · Reinette Chatre
The events needed to read memory bandwidth are discovered by iterating over every memory controller (iMC) within
/sys/bus/event_source/devices. Each iMC's PMU is assumed to have one event to measure read memory bandwidth that is represented by the sysfscas_count_readfile. The event's configuration is read from "cas_count_read" and stored as an element of imc_counters_config[] byread_from_imc_dir()that receives the index of the array where to store the configuration as argument.It is possible that an iMC's PMU may have more than one event that should be used to measure memory bandwidth.
Change semantics to not provide the index of the array to
read_from_imc_dir()but instead a pointer to the index. This enablesread_from_imc_dir()to store configurations for more than one event by incrementing the index to imc_counters_config[] itself.
Quoted 6 lines
Quoted 4 lines
Ensure that the same type is consistently used for the index as it is passed around during counter configuration.
Signed-off-by Reinette Chatre <[email protected]>Reviewed-by Zide Chen <[email protected]>
Thank you very much.
Reinette
Reinette Chatre
maintainerRe: [PATCH 3/8] selftests/resctrl: Support multiple events associated with iMC
In reply to Chen, ZideHi Zide,
Quoted 21 lines · Zide
Quoted 18 lines · Reinette Chatre
The resctrl selftests discover needed parameters to
perf_event_open()via sysfs. The PMU associated with every memory controller (iMC) is discovered via the/sys/bus/event_source/devices/uncore_imc_N/typefile while the read memory bandwidth event type and umask is discovered via/sys/bus/event_source/devices/uncore_imc_N/events/cas_count_read.Newer systems may have multiple events that expose read memory bandwidth. For example,
/sys/bus/event_source/devices/uncore_imc_N/events/cas_count_read_sch0/sys/bus/event_source/devices/uncore_imc_N/events/cas_count_read_sch1Support parsing of iMC PMU properties when the PMU may have multiple events to measure read memory bandwidth. The PMU only needs to be discovered once. Split the parsing of event details from actual PMU discovery in order to loop over all events associated with the PMU. Match all events with the
cas_count_readprefix instead of requiring there to be one file with that name.
Quoted 8 lines
Quoted 6 lines
Make the parsing code more robust. With strings passed around to create needed paths, use
snprintf()instead ofsprintf()to ensure there is always enough space to create the path. Ensure there is enough room in imc_counters_config[] before attempting to add an entry.Signed-off-by Reinette Chatre <[email protected]>Reviewed-by Zide Chen <[email protected]>
Thank you very much for taking a look at this work and especially thanks for creating the perf changes that they depend on to fix these tests.
Reinette