Skip to contentSkip to conversation

perf test: Make metric testing more robust.

PATCH Superseded Ian Rogers 2 participants 4 messages Aug 4, 2021 – Sep 16, 2021 1 message not in the archive perf events

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Ian Rogers

author reviewer
#0 lore raw

PATCH 1/1 perf test: Make metric testing more robust.

When testing metric expressions we fake counter values from 1 going upward. For some metrics this can yield negative values that are clipped to zero, and then cause divide by zero failures. A workaround for this case is to try a second time with counter values going in the opposite direction.

This case was seen in a metric like:
event1 / max(event2 - event3, 0)

---
 tools/perf/tests/pmu-events.c | 32 ++++++++++++++++++++++++++------
 1 file changed, 26 insertions(+), 6 deletions(-)
tools/perf/tests/pmu-events.c +26 -6
index b8aff8fb50d8..6c1cd58605c1 100644
@@ -600,8 +600,18 @@ static int test_parsing(void)
 			}
 
 			if (expr__parse(&result, &ctx, pe->metric_expr, 0)) {
-				expr_failure("Parse failed", map, pe);
-				ret++;
+				/*
+				 * Parsing failed, make numbers go from large to
+				 * small which can resolve divide by zero
+				 * issues.
+				 */
+				k = 1024;
+				hashmap__for_each_entry((&ctx.ids), cur, bkt)
+					expr__add_id_val(&ctx, strdup(cur->key), k--);
+				if (expr__parse(&result, &ctx, pe->metric_expr, 0)) {
+					expr_failure("Parse failed", map, pe);
+					ret++;
+				}
 			}
 			expr__ctx_clear(&ctx);
 		}
@@ -656,10 +666,20 @@ static int metric_parse_fake(const char *str)
 		}
 	}
 
-	if (expr__parse(&result, &ctx, str, 0))
-		pr_err("expr__parse failed\n");
-	else
-		ret = 0;
+	ret = 0;
+	if (expr__parse(&result, &ctx, str, 0)) {
+		/*
+		 * Parsing failed, make numbers go from large to small which can
+		 * resolve divide by zero issues.
+		 */
+		i = 1024;
+		hashmap__for_each_entry((&ctx.ids), cur, bkt)
+			expr__add_id_val(&ctx, strdup(cur->key), i--);
+		if (expr__parse(&result, &ctx, str, 0)) {
+			pr_err("expr__parse failed\n");
+			ret = -1;
+		}
+	}
 
 out:
 	expr__ctx_clear(&ctx);
Signature · 1 line
2.32.0.554.ge1b32706d8-goog
#1 lore raw
Quoted 8 lines · Ian Rogers

When testing metric expressions we fake counter values from 1 going upward. For some metrics this can yield negative values that are clipped to zero, and then cause divide by zero failures. A workaround for this case is to try a second time with counter values going in the opposite direction.

This case was seen in a metric like:

   event1 / max(event2 - event3, 0)

is this the standard method to make the metric evaluation fail when results are not as expected? In this example, event2 should be greater than event3 always. Dividing by max(x, 0) would seem a bit silly.

thanks,
John

Quoted 57 lines
---
  tools/perf/tests/pmu-events.c | 32 ++++++++++++++++++++++++++------
  1 file changed, 26 insertions(+), 6 deletions(-)
tools/perf/tests/pmu-events.c +26 -6
index b8aff8fb50d8..6c1cd58605c1 100644
@@ -600,8 +600,18 @@ static int test_parsing(void)
  			}
 
  			if (expr__parse(&result, &ctx, pe->metric_expr, 0)) {
-				expr_failure("Parse failed", map, pe);
-				ret++;
+				/*
+				 * Parsing failed, make numbers go from large to
+				 * small which can resolve divide by zero
+				 * issues.
+				 */
+				k = 1024;
+				hashmap__for_each_entry((&ctx.ids), cur, bkt)
+					expr__add_id_val(&ctx, strdup(cur->key), k--);
+				if (expr__parse(&result, &ctx, pe->metric_expr, 0)) {
+					expr_failure("Parse failed", map, pe);
+					ret++;
+				}
  			}
  			expr__ctx_clear(&ctx);
  		}
@@ -656,10 +666,20 @@ static int metric_parse_fake(const char *str)
  		}
  	}
 
-	if (expr__parse(&result, &ctx, str, 0))
-		pr_err("expr__parse failed\n");
-	else
-		ret = 0;
+	ret = 0;
+	if (expr__parse(&result, &ctx, str, 0)) {
+		/*
+		 * Parsing failed, make numbers go from large to small which can
+		 * resolve divide by zero issues.
+		 */
+		i = 1024;
+		hashmap__for_each_entry((&ctx.ids), cur, bkt)
+			expr__add_id_val(&ctx, strdup(cur->key), i--);
+		if (expr__parse(&result, &ctx, str, 0)) {
+			pr_err("expr__parse failed\n");
+			ret = -1;
+		}
+	}
 
  out:
  	expr__ctx_clear(&ctx);

A message that is not in the archive (sent off-list or not yet received): CAP-5=fWWvGi8gGfwGORF-4y7V6G_gCxY7=SX2zo_cyxKxTEf+A@mail.gmail.com

#3 lore raw
Quoted 17 lines · Ian Rogers

On Wed, Aug 4, 2021, 2:11 AM John Garry <[email protected] <mailto:[email protected]>> wrote:

On 04/08/2021 08:25, Ian Rogers wrote:
> When testing metric expressions we fake counter values from 1 going > upward. For some metrics this can yield negative values that are
clipped
> to zero, and then cause divide by zero failures. A workaround for
this
> case is to try a second time with counter values going in the
opposite
> direction.
>
> This case was seen in a metric like:
>    event1 / max(event2 - event3, 0)

Quoted 6 lines

is this the standard method to make the metric evaluation fail when results are not as expected? In this example, event2 should be greater than event3 always. Dividing by max(x, 0) would seem a bit silly.

I wouldn't say it was standard but it is in a metric a third party gave us.

I agree that making it more robust is a good thing. But masking bogus expressions isn't great. After all, we're here to find them :)

Quoted 2 lines

It would be possible to get the same test failure on more standard expressions, so it would be nice if these tests were more robust.

so something like this would fail also:
event1 / (event2 + event3 - 1 - event4)

assuming we have ascending values from 1 for event1. And this would seem a valid expression.

Anyway, it would be nice if we could reject max(0, x) and any divide by negative numbers, apart from your change.

Cheers,
john

Quoted 68 lines
     >
     > Signed-off-by: Ian Rogers <[email protected]
    <mailto:[email protected]>>
     > ---
     >   tools/perf/tests/pmu-events.c | 32 ++++++++++++++++++++++++++------
     >   1 file changed, 26 insertions(+), 6 deletions(-)
     >
     > diff --git a/tools/perf/tests/pmu-events.c
    b/tools/perf/tests/pmu-events.c
     > index b8aff8fb50d8..6c1cd58605c1 100644
     > --- a/tools/perf/tests/pmu-events.c
     > +++ b/tools/perf/tests/pmu-events.c
     > @@ -600,8 +600,18 @@ static int test_parsing(void)
     >                       }
     >
     >                       if (expr__parse(&result, &ctx,
    pe->metric_expr, 0)) {
     > -                             expr_failure("Parse failed", map, pe);
     > -                             ret++;
     > +                             /*
     > +                              * Parsing failed, make numbers go
    from large to
     > +                              * small which can resolve divide
    by zero
     > +                              * issues.
     > +                              */
     > +                             k = 1024;
     > +                             hashmap__for_each_entry((&ctx.ids),
    cur, bkt)
     > +                                     expr__add_id_val(&ctx,
    strdup(cur->key), k--);
     > +                             if (expr__parse(&result, &ctx,
    pe->metric_expr, 0)) {
     > +                                     expr_failure("Parse
    failed", map, pe);
     > +                                     ret++;
     > +                             }
     >                       }
     >                       expr__ctx_clear(&ctx);
     >               }
     > @@ -656,10 +666,20 @@ static int metric_parse_fake(const char *str)
     >               }
     >       }
     >
     > -     if (expr__parse(&result, &ctx, str, 0))
     > -             pr_err("expr__parse failed\n");
     > -     else
     > -             ret = 0;
     > +     ret = 0;
     > +     if (expr__parse(&result, &ctx, str, 0)) {
     > +             /*
     > +              * Parsing failed, make numbers go from large to
    small which can
     > +              * resolve divide by zero issues.
     > +              */
     > +             i = 1024;
     > +             hashmap__for_each_entry((&ctx.ids), cur, bkt)
     > +                     expr__add_id_val(&ctx, strdup(cur->key), i--);
     > +             if (expr__parse(&result, &ctx, str, 0)) {
     > +                     pr_err("expr__parse failed\n");
     > +                     ret = -1;
     > +             }
     > +     }
     >
     >   out:
     >       expr__ctx_clear(&ctx);
     >
author reviewer
#4 lore raw
Quoted 34 lines · John Garry
Quoted 18 lines · Ian Rogers

On Wed, Aug 4, 2021, 2:11 AM John Garry <[email protected] <mailto:[email protected]>> wrote:

On 04/08/2021 08:25, Ian Rogers wrote:
> When testing metric expressions we fake counter values from 1 going > upward. For some metrics this can yield negative values that are
clipped
> to zero, and then cause divide by zero failures. A workaround for
this
> case is to try a second time with counter values going in the
opposite
> direction.
>
> This case was seen in a metric like:
> event1 / max(event2 - event3, 0)

Quoted 7 lines

is this the standard method to make the metric evaluation fail when results are not as expected? In this example, event2 should be greater than event3 always. Dividing by max(x, 0) would seem a bit silly.

I wouldn't say it was standard but it is in a metric a third party gave us.

I agree that making it more robust is a good thing. But masking bogus expressions isn't great. After all, we're here to find them :)

Quoted 2 lines

It would be possible to get the same test failure on more standard expressions, so it would be nice if these tests were more robust.

Quoted 8 lines

so something like this would fail also:
event1 / (event2 + event3 - 1 - event4)

assuming we have ascending values from 1 for event1. And this would seem a valid expression.

Anyway, it would be nice if we could reject max(0, x) and any divide by negative numbers, apart from your change.

Thanks John, it'd be nice to have a tool to vet for bogus expressions. I think that's out of the scope of this change but we should bear it in mind. Would be nice to land this change if someone has time to review.

Thanks,
Ian

Quoted 73 lines

Cheers,
john

Quoted 68 lines
     >
     > Signed-off-by: Ian Rogers <[email protected]
    <mailto:[email protected]>>
     > ---
     >   tools/perf/tests/pmu-events.c | 32 ++++++++++++++++++++++++++------
     >   1 file changed, 26 insertions(+), 6 deletions(-)
     >
     > diff --git a/tools/perf/tests/pmu-events.c
    b/tools/perf/tests/pmu-events.c
     > index b8aff8fb50d8..6c1cd58605c1 100644
     > --- a/tools/perf/tests/pmu-events.c
     > +++ b/tools/perf/tests/pmu-events.c
     > @@ -600,8 +600,18 @@ static int test_parsing(void)
     >                       }
     >
     >                       if (expr__parse(&result, &ctx,
    pe->metric_expr, 0)) {
     > -                             expr_failure("Parse failed", map, pe);
     > -                             ret++;
     > +                             /*
     > +                              * Parsing failed, make numbers go
    from large to
     > +                              * small which can resolve divide
    by zero
     > +                              * issues.
     > +                              */
     > +                             k = 1024;
     > +                             hashmap__for_each_entry((&ctx.ids),
    cur, bkt)
     > +                                     expr__add_id_val(&ctx,
    strdup(cur->key), k--);
     > +                             if (expr__parse(&result, &ctx,
    pe->metric_expr, 0)) {
     > +                                     expr_failure("Parse
    failed", map, pe);
     > +                                     ret++;
     > +                             }
     >                       }
     >                       expr__ctx_clear(&ctx);
     >               }
     > @@ -656,10 +666,20 @@ static int metric_parse_fake(const char *str)
     >               }
     >       }
     >
     > -     if (expr__parse(&result, &ctx, str, 0))
     > -             pr_err("expr__parse failed\n");
     > -     else
     > -             ret = 0;
     > +     ret = 0;
     > +     if (expr__parse(&result, &ctx, str, 0)) {
     > +             /*
     > +              * Parsing failed, make numbers go from large to
    small which can
     > +              * resolve divide by zero issues.
     > +              */
     > +             i = 1024;
     > +             hashmap__for_each_entry((&ctx.ids), cur, bkt)
     > +                     expr__add_id_val(&ctx, strdup(cur->key), i--);
     > +             if (expr__parse(&result, &ctx, str, 0)) {
     > +                     pr_err("expr__parse failed\n");
     > +                     ret = -1;
     > +             }
     > +     }
     >
     >   out:
     >       expr__ctx_clear(&ctx);
     >