Skip to contentSkip to conversation

realtime: Fix the pass criterion of pi_perf test case.

PATCH In review Sripathi Kodi 2 participants 4 messages Jul 3, 2009 – Jul 7, 2009

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 realtime: Fix the pass criterion of pi_perf test case.

Hi,

The pass criterion in pi_perf test case is wrong. It compares the minimum amount of time taken by the low priority thread with the maximum amount of time taken by the high priority thread to calculate the PI delay. Obviously, these min and max don't necessarily happen in the same iteration, resulting in a number of false failures. The correct way is to compare the time taken by low and high priority threads in each iteration and then find the maximum delay experienced by high priority thread across the iterations. This patch implements the change. Additionally, this patch removes lock_wait_dat array, which is not needed anymore as well as makes a couple of messages easier to understand.

This patch changes the messages displayed by this test case slightly. They look like the following now:

Low prio thread started
High prio thread started
Busy 0 started
Busy 1 started
Busy 2 started
Busy 3 started
Time taken for high prio thread to get the lock once released by low prio thread
Min delay = 10 us
Max delay = 36 us
Average delay = 17.06 us
Standard Deviation = 8.11 us
Quantiles:
99.0% < 36
Criteria: High prio lock wait time < (Low prio lock held time + 200 us)
Result: PASS

I have tested this by running 1000 iterations of pi_perf test on a couple of machines.

Thanks,
Sripathi.

---

realtime: Fix the pass criterion of pi_perf test case.

testcases/realtime/func/pi_perf/pi_perf.c +12 -14
diff -uprN ltp_cvs_3jul_org/testcases/realtime/func/pi_perf/pi_perf.c ltp_cvs_3jul/testcases/realtime/func/pi_perf/pi_perf.c
@@ -1,6 +1,6 @@
 /******************************************************************************
  *
- *   Copyright © International Business Machines  Corp., 2007, 2008
+ *   Copyright © International Business Machines  Corp., 2007, 2008, 2009
  *
  *   This program is free software;  you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -36,6 +36,8 @@
  *
  * HISTORY
  *     2007-Nov-20:    Initial version by Sripathi Kodi <[email protected]>
+ *     2009-Jul-03:    Pass criteria corrected by Sripathi Kodi
+ *                                                      <[email protected]>
  *
  *****************************************************************************/
 
@@ -71,9 +73,9 @@ static unsigned int high_work_time = DEF
 static unsigned int busy_work_time;
 static int num_busy = -1;
 
-nsec_t low_unlock, high_get_lock;
+nsec_t low_unlock, max_pi_delay;
 
-stats_container_t lock_wait_dat, low_dat, cpu_delay_dat;
+stats_container_t low_dat, cpu_delay_dat;
 stats_container_t cpu_delay_hist;
 stats_quantiles_t cpu_delay_quantiles;
 
@@ -165,10 +167,9 @@ void * low_prio_thread(void *arg)
 
 void * high_prio_thread(void *arg)
 {
-	nsec_t high_start, high_spent, high_end;
+	nsec_t high_start, high_end, high_get_lock;
 	unsigned int i;
 
-	stats_container_init(&lock_wait_dat, iterations);
 	stats_container_init(&cpu_delay_dat, iterations);
 	stats_container_init(&cpu_delay_hist, HIST_BUCKETS);
 	stats_quantiles_init(&cpu_delay_quantiles, (int)log10(iterations));
@@ -184,14 +185,11 @@ void * high_prio_thread(void *arg)
 		high_start = rt_gettime();
 		pthread_mutex_lock(&lock);
 		high_end = rt_gettime();
-		high_spent = high_end - high_start;
 		high_get_lock = high_end - low_unlock;
 
 		busy_work_ms(high_work_time);
 		pthread_mutex_unlock(&lock);
 
-		lock_wait_dat.records[i].x = i;
-		lock_wait_dat.records[i].y = high_spent / NS_PER_US;
 		cpu_delay_dat.records[i].x = i;
 		cpu_delay_dat.records[i].y = high_get_lock / NS_PER_US;
 
@@ -206,14 +204,16 @@ void * high_prio_thread(void *arg)
 				"Latency (us)", "Samples", &cpu_delay_hist, "steps");
 
 	printf("Time taken for high prio thread to get the lock once released by low prio thread\n");
-	printf("Min wait time = %ld us\n", stats_min(&cpu_delay_dat));
-	printf("Max wait time = %ld us\n", stats_max(&cpu_delay_dat));
-	printf("Average wait time = %4.2f us\n", stats_avg(&cpu_delay_dat));
+	printf("Min delay = %ld us\n", stats_min(&cpu_delay_dat));
+	printf("Max delay = %ld us\n", stats_max(&cpu_delay_dat));
+	printf("Average delay = %4.2f us\n", stats_avg(&cpu_delay_dat));
 	printf("Standard Deviation = %4.2f us\n", stats_stddev(&cpu_delay_dat));
 	printf("Quantiles:\n");
 	stats_quantiles_calc(&cpu_delay_dat, &cpu_delay_quantiles);
 	stats_quantiles_print(&cpu_delay_quantiles);
 
+	max_pi_delay = stats_max(&cpu_delay_dat);
+
 	return NULL;
 }
 
@@ -261,13 +261,11 @@ int main(int argc, char *argv[])
 	}
 
 	join_threads();
-	printf("Low prio lock held time (min) = %ld us\n", stats_min(&low_dat));
-	printf("High prio lock wait time (max) = %ld us\n", stats_max(&lock_wait_dat));
 	printf("Criteria: High prio lock wait time < "
 			"(Low prio lock held time + %d us)\n", (int)pass_criteria);
 
 	ret = 0;
-	if (stats_max(&lock_wait_dat) > stats_min(&low_dat) + (int)pass_criteria)
+	if (max_pi_delay > pass_criteria)
 		ret = 1;
 
 	printf("Result: %s\n", ret ? "FAIL" : "PASS");

------------------------------------------------------------------------------
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

#1 lore raw
Quoted 36 lines · Sripathi Kodi

Hi,

The pass criterion in pi_perf test case is wrong. It compares the minimum amount of time taken by the low priority thread with the maximum amount of time taken by the high priority thread to calculate the PI delay. Obviously, these min and max don't necessarily happen in the same iteration, resulting in a number of false failures. The correct way is to compare the time taken by low and high priority threads in each iteration and then find the maximum delay experienced by high priority thread across the iterations. This patch implements the change. Additionally, this patch removes lock_wait_dat array, which is not needed anymore as well as makes a couple of messages easier to understand.

This patch changes the messages displayed by this test case slightly. They look like the following now:

Low prio thread started
High prio thread started
Busy 0 started
Busy 1 started
Busy 2 started
Busy 3 started
Time taken for high prio thread to get the lock once released by low prio thread
Min delay = 10 us
Max delay = 36 us
Average delay = 17.06 us
Standard Deviation = 8.11 us
Quantiles:
99.0% < 36
Criteria: High prio lock wait time < (Low prio lock held time + 200 us)
Result: PASS

I have tested this by running 1000 iterations of pi_perf test on a couple of machines.

Quoted 8 lines

Thanks,
Sripathi.

---

realtime: Fix the pass criterion of pi_perf test case.

Hi Sri,

Is this based on the latest snapshot. I get the following failures:

patching file testcases/realtime/func/pi_perf/pi_perf.c Hunk #1 FAILED at 1.

Can you please resend ?

Regards--
Subrata

Quoted 89 lines

diff -uprN ltp_cvs_3jul_org/testcases/realtime/func/pi_perf/pi_perf.c ltp_cvs_3jul/testcases/realtime/func/pi_perf/pi_perf.c

testcases/realtime/func/pi_perf/pi_perf.c +11 -12
@@ -1,6 +1,6 @@
 /******************************************************************************
  *
- *   Copyright © International Business Machines  Corp., 2007, 2008
+ *   Copyright © International Business Machines  Corp., 2007, 2008, 2009
  *
  *   This program is free software;  you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -36,6 +36,8 @@
  *
  * HISTORY
  *     2007-Nov-20:    Initial version by Sripathi Kodi <[email protected]>
+ *     2009-Jul-03:    Pass criteria corrected by Sripathi Kodi
+ *                                                      <[email protected]>
  *
  *****************************************************************************/
 
@@ -71,9 +73,9 @@ static unsigned int high_work_time = DEF
 static unsigned int busy_work_time;
 static int num_busy = -1;
 
-nsec_t low_unlock, high_get_lock;
+nsec_t low_unlock, max_pi_delay;
 
-stats_container_t lock_wait_dat, low_dat, cpu_delay_dat;
+stats_container_t low_dat, cpu_delay_dat;
 stats_container_t cpu_delay_hist;
 stats_quantiles_t cpu_delay_quantiles;
 
@@ -165,10 +167,9 @@ void * low_prio_thread(void *arg)
 
 void * high_prio_thread(void *arg)
 {
-	nsec_t high_start, high_spent, high_end;
+	nsec_t high_start, high_end, high_get_lock;
 	unsigned int i;
 
-	stats_container_init(&lock_wait_dat, iterations);
 	stats_container_init(&cpu_delay_dat, iterations);
 	stats_container_init(&cpu_delay_hist, HIST_BUCKETS);
 	stats_quantiles_init(&cpu_delay_quantiles, (int)log10(iterations));
@@ -184,14 +185,11 @@ void * high_prio_thread(void *arg)
 		high_start = rt_gettime();
 		pthread_mutex_lock(&lock);
 		high_end = rt_gettime();
-		high_spent = high_end - high_start;
 		high_get_lock = high_end - low_unlock;
 
 		busy_work_ms(high_work_time);
 		pthread_mutex_unlock(&lock);
 
-		lock_wait_dat.records[i].x = i;
-		lock_wait_dat.records[i].y = high_spent / NS_PER_US;
 		cpu_delay_dat.records[i].x = i;
 		cpu_delay_dat.records[i].y = high_get_lock / NS_PER_US;
 
@@ -206,14 +204,16 @@ void * high_prio_thread(void *arg)
 				"Latency (us)", "Samples", &cpu_delay_hist, "steps");
 
 	printf("Time taken for high prio thread to get the lock once released by low prio thread\n");
-	printf("Min wait time = %ld us\n", stats_min(&cpu_delay_dat));
-	printf("Max wait time = %ld us\n", stats_max(&cpu_delay_dat));
-	printf("Average wait time = %4.2f us\n", stats_avg(&cpu_delay_dat));
+	printf("Min delay = %ld us\n", stats_min(&cpu_delay_dat));
+	printf("Max delay = %ld us\n", stats_max(&cpu_delay_dat));
+	printf("Average delay = %4.2f us\n", stats_avg(&cpu_delay_dat));
 	printf("Standard Deviation = %4.2f us\n", stats_stddev(&cpu_delay_dat));
 	printf("Quantiles:\n");
 	stats_quantiles_calc(&cpu_delay_dat, &cpu_delay_quantiles);
 	stats_quantiles_print(&cpu_delay_quantiles);
 
+	max_pi_delay = stats_max(&cpu_delay_dat);
+
 	return NULL;
 }
 
@@ -261,13 +261,11 @@ int main(int argc, char *argv[])
 	}
 
 	join_threads();
-	printf("Low prio lock held time (min) = %ld us\n", stats_min(&low_dat));

testcases/realtime/func/pi_perf/pi_perf.c int main(int argc, char *argv[]) · line 271

Quoted 10 lines
testcases/realtime/func/pi_perf/pi_perf.c +1 -2
-	printf("High prio lock wait time (max) = %ld us\n", stats_max(&lock_wait_dat));
 	printf("Criteria: High prio lock wait time < "
 			"(Low prio lock held time + %d us)\n", (int)pass_criteria);
 
 	ret = 0;
-	if (stats_max(&lock_wait_dat) > stats_min(&low_dat) + (int)pass_criteria)
+	if (max_pi_delay > pass_criteria)
 		ret = 1;
 
 	printf("Result: %s\n", ret ? "FAIL" : "PASS");

------------------------------------------------------------------------------
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

author Acked-by Will Schmidt Acked-by Gowrishankar
#2 lore raw
Quoted 48 lines · On Monday 06 July 2009 21:20:49 Subrata Modak wrote:
Quoted 36 lines · Sripathi Kodi

Hi,

The pass criterion in pi_perf test case is wrong. It compares the minimum amount of time taken by the low priority thread with the maximum amount of time taken by the high priority thread to calculate the PI delay. Obviously, these min and max don't necessarily happen in the same iteration, resulting in a number of false failures. The correct way is to compare the time taken by low and high priority threads in each iteration and then find the maximum delay experienced by high priority thread across the iterations. This patch implements the change. Additionally, this patch removes lock_wait_dat array, which is not needed anymore as well as makes a couple of messages easier to understand.

This patch changes the messages displayed by this test case slightly. They look like the following now:

Low prio thread started
High prio thread started
Busy 0 started
Busy 1 started
Busy 2 started
Busy 3 started
Time taken for high prio thread to get the lock once released by low prio thread
Min delay = 10 us
Max delay = 36 us
Average delay = 17.06 us
Standard Deviation = 8.11 us
Quantiles:
99.0% < 36
Criteria: High prio lock wait time < (Low prio lock held time + 200 us)
Result: PASS

I have tested this by running 1000 iterations of pi_perf test on a couple of machines.

Quoted 8 lines

Thanks,
Sripathi.

---

realtime: Fix the pass criterion of pi_perf test case.

Hi Sri,

Quoted 7 lines

Is this based on the latest snapshot. I get the following failures:

patching file testcases/realtime/func/pi_perf/pi_perf.c Hunk #1 FAILED at 1.

Can you please resend ?

Sorry Subrata, the patch seems to be fine on my machine, but is going bad in my mail client. If you don't mind, I have attached the patch to this mail for now.

Thanks,
Sripathi.

--

realtime: Fix the pass criterion of pi_perf test case.

<snip>

testcases/realtime/func/pi_perf/pi_perf.c +12 -14
diff -uprN ltp_cvs_3jul_org/testcases/realtime/func/pi_perf/pi_perf.c ltp_cvs_3jul/testcases/realtime/func/pi_perf/pi_perf.c
@@ -1,6 +1,6 @@
 /******************************************************************************
  *
- *   Copyright © International Business Machines  Corp., 2007, 2008
+ *   Copyright © International Business Machines  Corp., 2007, 2008, 2009
  *
  *   This program is free software;  you can redistribute it and/or modify
  *   it under the terms of the GNU General Public License as published by
@@ -36,6 +36,8 @@
  *
  * HISTORY
  *     2007-Nov-20:    Initial version by Sripathi Kodi <[email protected]>
+ *     2009-Jul-03:    Pass criteria corrected by Sripathi Kodi
+ *                                                      <[email protected]>
  *
  *****************************************************************************/
 
@@ -71,9 +73,9 @@ static unsigned int high_work_time = DEF
 static unsigned int busy_work_time;
 static int num_busy = -1;
 
-nsec_t low_unlock, high_get_lock;
+nsec_t low_unlock, max_pi_delay;
 
-stats_container_t lock_wait_dat, low_dat, cpu_delay_dat;
+stats_container_t low_dat, cpu_delay_dat;
 stats_container_t cpu_delay_hist;
 stats_quantiles_t cpu_delay_quantiles;
 
@@ -165,10 +167,9 @@ void * low_prio_thread(void *arg)
 
 void * high_prio_thread(void *arg)
 {
-	nsec_t high_start, high_spent, high_end;
+	nsec_t high_start, high_end, high_get_lock;
 	unsigned int i;
 
-	stats_container_init(&lock_wait_dat, iterations);
 	stats_container_init(&cpu_delay_dat, iterations);
 	stats_container_init(&cpu_delay_hist, HIST_BUCKETS);
 	stats_quantiles_init(&cpu_delay_quantiles, (int)log10(iterations));
@@ -184,14 +185,11 @@ void * high_prio_thread(void *arg)
 		high_start = rt_gettime();
 		pthread_mutex_lock(&lock);
 		high_end = rt_gettime();
-		high_spent = high_end - high_start;
 		high_get_lock = high_end - low_unlock;
 
 		busy_work_ms(high_work_time);
 		pthread_mutex_unlock(&lock);
 
-		lock_wait_dat.records[i].x = i;
-		lock_wait_dat.records[i].y = high_spent / NS_PER_US;
 		cpu_delay_dat.records[i].x = i;
 		cpu_delay_dat.records[i].y = high_get_lock / NS_PER_US;
 
@@ -206,14 +204,16 @@ void * high_prio_thread(void *arg)
 				"Latency (us)", "Samples", &cpu_delay_hist, "steps");
 
 	printf("Time taken for high prio thread to get the lock once released by low prio thread\n");
-	printf("Min wait time = %ld us\n", stats_min(&cpu_delay_dat));
-	printf("Max wait time = %ld us\n", stats_max(&cpu_delay_dat));
-	printf("Average wait time = %4.2f us\n", stats_avg(&cpu_delay_dat));
+	printf("Min delay = %ld us\n", stats_min(&cpu_delay_dat));
+	printf("Max delay = %ld us\n", stats_max(&cpu_delay_dat));
+	printf("Average delay = %4.2f us\n", stats_avg(&cpu_delay_dat));
 	printf("Standard Deviation = %4.2f us\n", stats_stddev(&cpu_delay_dat));
 	printf("Quantiles:\n");
 	stats_quantiles_calc(&cpu_delay_dat, &cpu_delay_quantiles);
 	stats_quantiles_print(&cpu_delay_quantiles);
 
+	max_pi_delay = stats_max(&cpu_delay_dat);
+
 	return NULL;
 }
 
@@ -261,13 +261,11 @@ int main(int argc, char *argv[])
 	}
 
 	join_threads();
-	printf("Low prio lock held time (min) = %ld us\n", stats_min(&low_dat));
-	printf("High prio lock wait time (max) = %ld us\n", stats_max(&lock_wait_dat));
 	printf("Criteria: High prio lock wait time < "
 			"(Low prio lock held time + %d us)\n", (int)pass_criteria);
 
 	ret = 0;
-	if (stats_max(&lock_wait_dat) > stats_min(&low_dat) + (int)pass_criteria)
+	if (max_pi_delay > pass_criteria)
 		ret = 1;
 
 	printf("Result: %s\n", ret ? "FAIL" : "PASS");

------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry

_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

#3 lore raw
Quoted 55 lines · Sripathi Kodi
Quoted 54 lines · On Monday 06 July 2009 21:20:49 Subrata Modak wrote:
Quoted 36 lines · Sripathi Kodi

Hi,

The pass criterion in pi_perf test case is wrong. It compares the minimum amount of time taken by the low priority thread with the maximum amount of time taken by the high priority thread to calculate the PI delay. Obviously, these min and max don't necessarily happen in the same iteration, resulting in a number of false failures. The correct way is to compare the time taken by low and high priority threads in each iteration and then find the maximum delay experienced by high priority thread across the iterations. This patch implements the change. Additionally, this patch removes lock_wait_dat array, which is not needed anymore as well as makes a couple of messages easier to understand.

This patch changes the messages displayed by this test case slightly. They look like the following now:

Low prio thread started
High prio thread started
Busy 0 started
Busy 1 started
Busy 2 started
Busy 3 started
Time taken for high prio thread to get the lock once released by low prio thread
Min delay = 10 us
Max delay = 36 us
Average delay = 17.06 us
Standard Deviation = 8.11 us
Quantiles:
99.0% < 36
Criteria: High prio lock wait time < (Low prio lock held time + 200 us)
Result: PASS

I have tested this by running 1000 iterations of pi_perf test on a couple of machines.

Quoted 8 lines

Thanks,
Sripathi.

---

realtime: Fix the pass criterion of pi_perf test case.

Hi Sri,

Is this based on the latest snapshot. I get the following failures:

patching file testcases/realtime/func/pi_perf/pi_perf.c Hunk #1 FAILED at 1.

Quoted 8 lines
Quoted 1 line

Can you please resend ?

Sorry Subrata, the patch seems to be fine on my machine, but is going bad in my mail client. If you don't mind, I have attached the patch to this mail for now.

Thanks,
Sripathi.

Thanks.

Regards--
Subrata

Quoted 10 lines

--

realtime: Fix the pass criterion of pi_perf test case.

<snip>

------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time, vendors submitting new applications to BlackBerry App World(TM) will have the opportunity to enter the BlackBerry Developer Challenge. See full prize details at: http://p.sf.net/sfu/blackberry
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list