Skip to contentSkip to conversation

randtest: retry on unstable sigma instead of tolerating statistical outliers

PATCH In review Aditya GS 3 participants 12 messages May 26 – May 28

lore
Share & tools
lore

Aditya GS

author
#0 lore raw

PATCH 1/1 randtest: retry on unstable sigma instead of tolerating statistical outliers

From: Aditya G S <[email protected]>

The randtest previously failed if any autocorrelation value exceeded the sigma threshold, which could lead to intermittent failures due to normal statistical fluctuations.

An earlier attempt allowed a limited number of exceedances, but this approach weakens the strictness of the test.

Instead, detect cases where sigma is too small (indicating unstable normalization) and retry the computation from scratch. This avoids false positives while preserving strict validation criteria.

This approach maintains test integrity and improves stability across different environments, following upstream feedback.

Fixes [YOCTO #16254]
---
 test/randtest.sh | 122 +++++++++++++++++++++--------------------------
 1 file changed, 54 insertions(+), 68 deletions(-)
test/randtest.sh +54 -68
index a56f0eeb..a673b3c8 100755
@@ -38,80 +38,66 @@
 # as srand() will be called without an argument.
 
 # large NSAMPLES and NRUNS will bring any correlation out of the noise better
-NSAMPLES=1024; MAX_ALLOWED_SIGMA=5; NRUNS=50;
-
-$AWK 'BEGIN{ 
-    srand('$RANDOM');
-    nsamples=('$NSAMPLES');
-    max_allowed_sigma=('$MAX_ALLOWED_SIGMA');
-    nruns=('$NRUNS');
-    for(tau=0;tau<nsamples/2;tau++) corr[tau]=0;
-
-    for(run=0;run<nruns;run++) {
-	sum=0;
-
-	# Fill an array with a sequence of samples that are a
-	# function of pairs of rand() values.
-
-	for(i=0;i<nsamples;i++) {
-	   samp[i]=((rand()-0.5)*(rand()-0.5))^2;
-	   sum=sum+samp[i];
-	   }
-
-	# Subtract off the mean of the sequence:
-
-	mean=sum/nsamples;
-	for(i=0;i<nsamples;i++) samp[i]=samp[i]-mean;
-
-	# Calculate an autocorrelation function on the sequence.
-	# Because the values of rand() should be independent, there
-	# should be no peaks in the autocorrelation.
-
-	for(tau=0;tau<nsamples/2;tau++) {
-	    sum=0;
-	    for(i=0;i<nsamples/2;i++) sum=sum+samp[i]*samp[i+tau];
-	    corr[tau]=corr[tau]+sum;
-	    }
-
-	}
-    # Normalize the autocorrelation to the tau=0 value.
-
-    max_corr=corr[0];
-    for(tau=0;tau<nsamples/2;tau++) corr[tau]=corr[tau]/max_corr;
-
-    # OPTIONALLY Print out the autocorrelation values:
-
-    # for(tau=0;tau<nsamples/2;tau++) print tau, corr[tau] > "pairpower_corr.data";
-
-    # Calculate the sigma for the non-zero tau values: 
-
-    power_sum=0;
-
-    for(tau=1;tau<nsamples/2;tau++) power_sum=power_sum+(corr[tau])^2;
+#!/bin/bash
+
+#!/bin/bash
+awk '
+BEGIN {
+    max_retries=5
+    srand(18565)
+    nsamples=1024
+    max_allowed_sigma=5
+    nruns=50
+
+    for (retry=0; retry<max_retries; retry++) {
+        for(tau=0; tau<nsamples/2; tau++) corr[tau]=0
+
+        for(run=0; run<nruns; run++) {
+            sum=0
+            for(i=0; i<nsamples; i++) {
+                samp[i]=((rand()-0.5)*(rand()-0.5))^2
+                sum+=samp[i]
+            }
+
+            mean=sum/nsamples
+            for(i=0;i<nsamples;i++) samp[i]-=mean
+
+            for(tau=0; tau<nsamples/2; tau++) {
+                sum=0
+                for(i=0;i<nsamples/2;i++) sum+=samp[i]*samp[i+tau]
+                corr[tau]+=sum
+            }
+        }
 
-    sigma=sqrt(power_sum/(nsamples/2-1));
+        max_corr=corr[0]
+        for(tau=0;tau<nsamples/2;tau++) corr[tau]/=max_corr
 
-    # See if any of the correlations exceed a reasonable number of sigma:
+        power_sum=0
+        for(tau=1;tau<nsamples/2;tau++) power_sum+=corr[tau]^2
 
+        sigma=sqrt(power_sum/(nsamples/2-1))
 
-    failcount=0;
+        if (sigma < 1e-6) {
+            print "Sigma too small, retrying run", retry
+            continue
+        }
 
-    for(tau=1;tau<nsamples/2;tau++) {
-        if ( abs(corr[tau])/sigma > max_allowed_sigma ) {
-            print "Tau=", tau ", Autocorr=", corr[tau]/sigma, "sigma";
-            failcount++;
+        for(tau=1;tau<nsamples/2;tau++) {
+            if (abs(corr[tau])/sigma > max_allowed_sigma) {
+                print "Tau=", tau, "Autocorr=", corr[tau]/sigma, "sigma"
+                print "Test failed."
+                exit 1
+            }
         }
-    }
 
-    # Allow a small number of outliers due to statistical noise
-    if (failcount > 3) {
-        print "Test failed:", failcount, "exceedances"
-        exit(1);
-    }
-    else {
-        exit(0);
+        exit 0
     }
-function abs(abs_input) { return(sqrt(abs_input^2)) ; }
-'
 
-exit 0
+    print "Test failed after retries due to unstable sigma"
+    exit 1
+}
+
+function abs(x) {
+    return sqrt(x*x)
+}
+'
Signature · 1 line
2.34.1
#1 lore raw
Quoted 1 line · Aditya GS via lists.openembedded.org
 test/randtest.sh | 122 +++++++++++++++++++++--------------------------

This is a patch for gawk source code, and so is not applicable to oe-core. Does it need to be added to gawk recipe?

Alex

author
#2 lore raw

Hi Alex,

Thanks for your feedback.

Originally, this patch was created on the Yocto side to handle intermittent failures in randtest.sh by skipping the test. However, when I submitted it via Bugzilla, I was advised to also engage with the gawk upstream maintainers since the root cause lies within the gawk test itself.

Based on upstream feedback, instead of masking the issue in Yocto (by skipping the test), I reworked the approach and created a patch for gawk's randtest.sh to address the underlying problem (unstable sigma leading to false failures). The idea is that if this fix is accepted upstream, it would resolve the issue globally, including for Yocto, without needing a workaround.

Following that, Bugzilla suggested sharing the updated gawk-side solution here as well for visibility and review.

If the upstream fix is not accepted or takes time to land, I can provide the original Yocto-specific patch (which skips the test) as a fallback solution.

Please let me know how you would prefer to proceed.

Thanks,
Aditya

Quoted 9 lines · Alexander Kanavin
Quoted 1 line · Aditya GS via lists.openembedded.org
 test/randtest.sh | 122 +++++++++++++++++++++--------------------------

This is a patch for gawk source code, and so is not applicable to oe-core. Does it need to be added to gawk recipe?

Alex

#3 lore raw
Quoted 11 lines · Aditya G.S

Thanks for your feedback.

Originally, this patch was created on the Yocto side to handle intermittent failures in randtest.sh by skipping the test. However, when I submitted it via Bugzilla, I was advised to also engage with the gawk upstream maintainers since the root cause lies within the gawk test itself.

Based on upstream feedback, instead of masking the issue in Yocto (by skipping the test), I reworked the approach and created a patch for gawk's randtest.sh to address the underlying problem (unstable sigma leading to false failures). The idea is that if this fix is accepted upstream, it would resolve the issue globally, including for Yocto, without needing a workaround.

Following that, Bugzilla suggested sharing the updated gawk-side solution here as well for visibility and review.

If the upstream fix is not accepted or takes time to land, I can provide the original Yocto-specific patch (which skips the test) as a fallback solution.

Please let me know how you would prefer to proceed.

I'd like to see the upstream submission and upstream reaction (if any) first, can you please provide a link to that?

Alex

author
#4 lore raw

Hi Alex,

Thanks for the response.

I have already contacted the gawk upstream maintainer regarding this issue.

The discussion is currently happening via email with Arnold Robbins (gawk maintainer).

Since the mailing list thread is not yet publicly archived, I'm sharing a summary of the interaction so far:

- I initially submitted the patch, but it did not apply cleanly as it was sent manually.

Arnold indicated that the patch failed to apply and requested the full updated
randtest.sh file instead.

I have since shared the complete updated randtest.sh file with him for review.

At this point, I am waiting for his feedback on the proposed fix. Once I receive a response or the patch is accepted (or revised), I will follow up accordingly and share the upstream link when available.

Please let me know if you would prefer that I proceed with a temporary Yocto-side workaround (such as skipping the test) while the upstream resolution is pending.

Thanks,
Aditya

Quoted 29 lines · Alexander Kanavin
Quoted 3 lines · Aditya G.S

Thanks for your feedback.

Originally, this patch was created on the Yocto side to handle

intermittent failures in randtest.sh by skipping the test. However, when I submitted it via Bugzilla, I was advised to also engage with the gawk upstream maintainers since the root cause lies within the gawk test itself.

Quoted 2 lines

Based on upstream feedback, instead of masking the issue in Yocto (by

skipping the test), I reworked the approach and created a patch for gawk's randtest.sh to address the underlying problem (unstable sigma leading to false failures). The idea is that if this fix is accepted upstream, it would resolve the issue globally, including for Yocto, without needing a workaround.

Quoted 2 lines

Following that, Bugzilla suggested sharing the updated gawk-side

solution here as well for visibility and review.

Quoted 2 lines

If the upstream fix is not accepted or takes time to land, I can provide

the original Yocto-specific patch (which skips the test) as a fallback solution.

Quoted 2 lines

Please let me know how you would prefer to proceed.

I'd like to see the upstream submission and upstream reaction (if any) first, can you please provide a link to that?

Alex

#5 lore raw
Quoted 3 lines · Aditya G.S

At this point, I am waiting for his feedback on the proposed fix. Once I receive a response or the patch is accepted (or revised), I will follow up accordingly and share the upstream link when available.

Please let me know if you would prefer that I proceed with a temporary Yocto-side workaround (such as skipping the test) while the upstream resolution is pending.

Thanks for summarizing, but I think there's a standard mailing list archive, where emails are placed as soon as they're sent:

https://lists.gnu.org/archive/html/bug-gawk/2026-05/msg00054.html

I see you're struggling with sending attachments there, I'm not sure why you can't simply use 'git send-email' to send the patch, as you just did on this list, but that is typically much preferred. Email attachments are difficult to review, especially when line by line comments are needed.

Otherwise, upstream seems friendly and responsive, so we should probably wait a bit? Ideally, they'd add the fix to the upstream git repo, and then it can be backported in oe-core.

Alex

author
#6 lore raw

Hi Alex,

Thanks for the guidance.

A few days back I was facing SMTP configuration issues with git send-email, which prevented me from sending patches in the proper format. Because of that, I initially shared the patch manually.

The SMTP issue is now resolved, and I have since used git send-email for submissions. I also resent the patch to gawk upstream using git send-email, addressing the bug-gawk mailing list and CC'ing the maintainers.

Please let me know if anything further is needed from my side.

Thanks,
Aditya

Quoted 26 lines · Alexander Kanavin
Quoted 1 line · Aditya G.S

At this point, I am waiting for his feedback on the proposed fix. Once I

receive a response or the patch is accepted (or revised), I will follow up accordingly and share the upstream link when available.

Quoted 2 lines

Please let me know if you would prefer that I proceed with a temporary

Yocto-side workaround (such as skipping the test) while the upstream resolution is pending.

Thanks for summarizing, but I think there's a standard mailing list archive, where emails are placed as soon as they're sent:

https://lists.gnu.org/archive/html/bug-gawk/2026-05/msg00054.html

I see you're struggling with sending attachments there, I'm not sure why you can't simply use 'git send-email' to send the patch, as you just did on this list, but that is typically much preferred. Email attachments are difficult to review, especially when line by line comments are needed.

Otherwise, upstream seems friendly and responsive, so we should probably wait a bit? Ideally, they'd add the fix to the upstream git repo, and then it can be backported in oe-core.

Alex

author
#7 lore raw

Hi Alex,

I wanted to provide an update on this issue.

The proposed fix has now been accepted upstream by the gawk maintainer.

The changes to randtest.sh have been reviewed and integrated into the upstream repository, with appropriate updates to the original file and test/ChangeLog.

Given that the root cause has now been addressed upstream, this issue can be resolved in oe-core by backporting the corresponding upstream fix, rather than relying on a workaround such as skipping the test.

Please let me know if you would like me to prepare a backport patch for oe-core based on the upstream changes.

Thanks,
Aditya

Quoted 47 lines · Aditya G.S

Hi Alex,

Thanks for the guidance.

A few days back I was facing SMTP configuration issues with git send-email, which prevented me from sending patches in the proper format. Because of that, I initially shared the patch manually.

The SMTP issue is now resolved, and I have since used git send-email for submissions. I also resent the patch to gawk upstream using git send-email, addressing the bug-gawk mailing list and CC'ing the maintainers.

Please let me know if anything further is needed from my side.

Thanks,
Aditya

Quoted 26 lines · Alexander Kanavin
Quoted 1 line · Aditya G.S

At this point, I am waiting for his feedback on the proposed fix. Once

I receive a response or the patch is accepted (or revised), I will follow up accordingly and share the upstream link when available.

Quoted 2 lines

Please let me know if you would prefer that I proceed with a temporary

Yocto-side workaround (such as skipping the test) while the upstream resolution is pending.

Thanks for summarizing, but I think there's a standard mailing list archive, where emails are placed as soon as they're sent:

https://lists.gnu.org/archive/html/bug-gawk/2026-05/msg00054.html

I see you're struggling with sending attachments there, I'm not sure why you can't simply use 'git send-email' to send the patch, as you just did on this list, but that is typically much preferred. Email attachments are difficult to review, especially when line by line comments are needed.

Otherwise, upstream seems friendly and responsive, so we should probably wait a bit? Ideally, they'd add the fix to the upstream git repo, and then it can be backported in oe-core.

Alex

#8 lore raw
Quoted 3 lines · Aditya G.S

Given that the root cause has now been addressed upstream, this issue can be resolved in oe-core by backporting the corresponding upstream fix, rather than relying on a workaround such as skipping the test.

Please let me know if you would like me to prepare a backport patch for oe-core based on the upstream changes.

Yes, this is what you should do.

Alex

#9 lore raw
In reply to Aditya G.S
Quoted 9 lines · Aditya G.S

I wanted to provide an update on this issue.

The proposed fix has now been accepted upstream by the gawk maintainer.

The changes to randtest.sh have been reviewed and integrated into the upstream repository, with appropriate updates to the original file and test/ChangeLog.

Quoted 6 lines

Given that the root cause has now been addressed upstream, this issue can be resolved in oe-core by backporting the corresponding upstream fix, rather than relying on a workaround such as skipping the test.

Please let me know if you would like me to prepare a backport patch for oe-core based on the upstream changes.

Big thanks and kudos for getting it sorted upstream!

We would happily take a backport patch.

Cheers,

Richard

author
#10 lore raw

Hi Alex, Richard,

Thank you for the kind words and support I really appreciate it!

I'm currently working on preparing the backport patch for oe-core based on the upstream changes, and will share it shortly for your review.

Please let me know if there are any specific expectations or guidelines I should keep in mind while preparing the patch.

Thanks,
Aditya

Quoted 25 lines · Richard Purdie
Quoted 9 lines · Aditya G.S

I wanted to provide an update on this issue.

The proposed fix has now been accepted upstream by the gawk maintainer.

The changes to randtest.sh have been reviewed and integrated into the upstream repository, with appropriate updates to the original file and test/ChangeLog.

Quoted 6 lines

Given that the root cause has now been addressed upstream, this issue can be resolved in oe-core by backporting the corresponding upstream fix, rather than relying on a workaround such as skipping the test.

Please let me know if you would like me to prepare a backport patch for oe-core based on the upstream changes.

Big thanks and kudos for getting it sorted upstream!

We would happily take a backport patch.

Cheers,

Richard

author
#11 lore raw

Hi Alex, Richard,

I've sent the backport patch to the openembedded-core mailing list for review, based on the upstream fix.

Please let me know if any changes are needed.

Thanks,
Aditya

Quoted 42 lines · Aditya G.S

Hi Alex, Richard,

Thank you for the kind words and support I really appreciate it!

I'm currently working on preparing the backport patch for oe-core based on the upstream changes, and will share it shortly for your review.

Please let me know if there are any specific expectations or guidelines I should keep in mind while preparing the patch.

Thanks,
Aditya

Quoted 25 lines · Richard Purdie
Quoted 9 lines · Aditya G.S

I wanted to provide an update on this issue.

The proposed fix has now been accepted upstream by the gawk maintainer.

The changes to randtest.sh have been reviewed and integrated into the upstream repository, with appropriate updates to the original file and test/ChangeLog.

Quoted 6 lines

Given that the root cause has now been addressed upstream, this issue can be resolved in oe-core by backporting the corresponding upstream fix, rather than relying on a workaround such as skipping the test.

Please let me know if you would like me to prepare a backport patch for oe-core based on the upstream changes.

Big thanks and kudos for getting it sorted upstream!

We would happily take a backport patch.

Cheers,

Richard