Skip to contentSkip to conversation

WANTED: xfstests results in different architectures

RFC PATCH In review Chandra Seetharaman 5 participants 12 messages Jun 18, 2013 – Jun 20, 2013

lore
Share & tools
lore
#0 lore raw

Hello All,

Couple of weeks backs we had a discussion in xfs meeting to collect xfstests results. I volunteered to collect xfstests results from different architectures and upload to XFS.org.

I can run and get the results for x86_64 and ppc64. If anyone has other architectures that they can run the tests on and provide me the results, I will filter them an upload to XFS.org.

Here is what I think would be of value to provide along with the results (others, please feel free to add more to the list for the results to be more useful)
- Architecture of the system
- Configuration - memory size and number of procs
- Filesystem sizes
- Commit ID of the kernel
- which git tree (XFS git tree or Linus's)
- xfsprogs version (or commit ID if from the git tree)

Thanks for your help in advance.

Regards,

Chandra

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 9 lines · Chandra Seetharaman

Hello All,

Couple of weeks backs we had a discussion in xfs meeting to collect xfstests results. I volunteered to collect xfstests results from different architectures and upload to XFS.org.

I can run and get the results for x86_64 and ppc64. If anyone has other architectures that they can run the tests on and provide me the results, I will filter them an upload to XFS.org.

How are you going to filter and display them on xfs.org? Should the scripts to do this be part of xfstests?

FWIW, without a database of results that users can use to filter the test results themselves, it will become unmanageable very quickly...

BTW, from my notes from the 2012 LSFMM XFs get-together, there are these line items related to exactly this:

----
- Public repository of test results so we can better track failures
- Look into resurrecting old ASG xfstests results repository and web iquery interface (Ben)
- host on oss.sgi.com.
- script to run xfstests and produce publishable output (Ben)
----

Ben, did you ever start to look into this?

Quoted 4 lines

Here is what I think would be of value to provide along with the results (others, please feel free to add more to the list for the results to be more useful)
- Architecture of the system

- base distro (e.g. /etc/release).

Quoted 1 line

- Configuration - memory size and number of procs

I think that all the info that we ask people to report in bug reports would be a good start....

Quoted 1 line

- Filesystem sizes

More useful is the MKFS_OPTIONS and MOUNT_OPTIONS used to run the tests, as that tells us how much non-default test coverage we are getting. i.e. default testing or something different.

Quoted 1 line

- Commit ID of the kernel

Not useful for kernels built with local, non-public changes, which is generally 100% of the kernels and userspace packages I test with.

Quoted 2 lines

- which git tree (XFS git tree or Linus's)
- xfsprogs version (or commit ID if from the git tree)

Same as for the kernel - base version is probably all that is useful here.

You'd probably also want to capture the console output indicating test runtimes and why certain tests weren't run.

If you create a pristine $RESULTS_DIR and output all the information you want to gather into it, then it will be trivial for users to send information onwards. Providing a command line parameter that generates a unique results directory and then packages the results up into a tarball would be a great start. We'd then have a single file that can be sent up a central point with all the test results available. We could even do all the filtering/processing before upload.

IOWs, the idea behind $RESULTS_DIR is to make this sort of scripted test result gathering simple to do....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#2 lore raw
Quoted 17 lines · Dave Chinner

How are you going to filter and display them on xfs.org? Should the scripts to do this be part of xfstests?

FWIW, without a database of results that users can use to filter the test results themselves, it will become unmanageable very quickly...

BTW, from my notes from the 2012 LSFMM XFs get-together, there are these line items related to exactly this:

----
- Public repository of test results so we can better track failures
- Look into resurrecting old ASG xfstests results repository and web iquery interface (Ben)
- host on oss.sgi.com.
- script to run xfstests and produce publishable output (Ben)
----

Quoted 1 line

Ben, did you ever start to look into this?

That became my work, not Ben's.

The current proposal is to gather architecture, distro, memory size, number of procs, characteristics of the storage device (size, performance, type, xfs_info output), xfsprogs version numbers, the time it takes for each test to run, the date & time it was run, and the hostname of the system.

The plan was to output that information into a database which we could add benchmark information into as well with an eye to detecting performance regressions.

(I felt, for example, that the recent discussion of CRC work and CPU usage could have been helped by that.)

The last stage was to write a script to upload test run data which oss would be able to publish.

Some of that work is done. I need to get back around to it.

-Phil

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
In reply to Dave Chinner

Dave, Chandra,

Quoted 26 lines · Dave Chinner
Quoted 7 lines · Chandra Seetharaman

Couple of weeks backs we had a discussion in xfs meeting to collect xfstests results. I volunteered to collect xfstests results from different architectures and upload to XFS.org.

I can run and get the results for x86_64 and ppc64. If anyone has other architectures that they can run the tests on and provide me the results, I will filter them an upload to XFS.org.

How are you going to filter and display them on xfs.org? Should the scripts to do this be part of xfstests?

FWIW, without a database of results that users can use to filter the test results themselves, it will become unmanageable very quickly...

BTW, from my notes from the 2012 LSFMM XFs get-together, there are these line items related to exactly this:

----
- Public repository of test results so we can better track failures
- Look into resurrecting old ASG xfstests results repository and web iquery interface (Ben)
- host on oss.sgi.com.
- script to run xfstests and produce publishable output (Ben)
----

Quoted 1 line

Ben, did you ever start to look into this?

I did. It is still pretty rough. I'll post it in reply here.

My design criteria as best as I can remember...

1) not filesystem specific

2) capture the actual results

3) simple, lightweight, very little setup

4) compatible with something like hangcheck so you can figure out when a machine has hung or crashed

5) useable for an individual developer

6) useable for a maintainer

7) not web centric

8) uses tools that are likely to be installed with the base distro

9) includes what you're testing, along with the results.

10) upload target can be easily changed.

11) (not implemented) perl scripts to parse the output.

...and so on. I have no objection to adding databases and web stuff on the backend.

Quoted 18 lines
Quoted 4 lines

Here is what I think would be of value to provide along with the results (others, please feel free to add more to the list for the results to be more useful)
- Architecture of the system

- base distro (e.g. /etc/release).

Quoted 1 line

- Configuration - memory size and number of procs

I think that all the info that we ask people to report in bug reports would be a good start....

Quoted 1 line

- Filesystem sizes

More useful is the MKFS_OPTIONS and MOUNT_OPTIONS used to run the tests, as that tells us how much non-default test coverage we are getting. i.e. default testing or something different.

Quoted 5 lines
Quoted 1 line

- Commit ID of the kernel

Not useful for kernels built with local, non-public changes, which is generally 100% of the kernels and userspace packages I test with.

You're using a guilt workflow, right? I think it could be useful if you grab the commit id, and then a list of all the changes atop that. I find this useful with my quilt workflow.

Quoted 8 lines
Quoted 2 lines

- which git tree (XFS git tree or Linus's)
- xfsprogs version (or commit ID if from the git tree)

Same as for the kernel - base version is probably all that is useful here.

You'd probably also want to capture the console output indicating test runtimes and why certain tests weren't run.

Agreed, gathering runtimes and .notrun is worth doing.

Quoted 11 lines

If you create a pristine $RESULTS_DIR and output all the information you want to gather into it, then it will be trivial for users to send information onwards. Providing a command line parameter that generates a unique results directory and then packages the results up into a tarball would be a great start. We'd then have a single file that can be sent up a central point with all the test results available. We could even do all the filtering/processing before upload.

IOWs, the idea behind $RESULTS_DIR is to make this sort of scripted test result gathering simple to do....

Posting results as you go has certain benefits, like hangcheck.

-Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw
In reply to Dave Chinner

On Wed, 2013-06-19 at 09:34 +1000, Dave Chinner wrote:

Hi Dave,

Quoted 7 lines · Chandra Seetharaman
Quoted 6 lines

Hello All,

Couple of weeks backs we had a discussion in xfs meeting to collect xfstests results. I volunteered to collect xfstests results from different architectures and upload to XFS.org.

Quoted 6 lines
Quoted 3 lines

I can run and get the results for x86_64 and ppc64. If anyone has other architectures that they can run the tests on and provide me the results, I will filter them an upload to XFS.org.

How are you going to filter and display them on xfs.org? Should the scripts to do this be part of xfstests?

I wasn't thinking of very elaborate filtering of all the results submitted by all variants of kernel/xfsprogs.

My thinking was very simple:
- get test results based on publicly available git tree commit IDs
- show the commit ID, and the failures seen in a specific arch.

This would help when anybody runs xfstests and sees a failure with any newer code, they would know if it is a regression or already seen.

But, looks like there is more elaborate work in progress. I will sync up with Ben and Phil to see how to help.

Quoted 59 lines

FWIW, without a database of results that users can use to filter the test results themselves, it will become unmanageable very quickly...

BTW, from my notes from the 2012 LSFMM XFs get-together, there are these line items related to exactly this:

----
- Public repository of test results so we can better track failures
- Look into resurrecting old ASG xfstests results repository and web iquery interface (Ben)
- host on oss.sgi.com.
- script to run xfstests and produce publishable output (Ben)
----

Ben, did you ever start to look into this?

Quoted 4 lines

Here is what I think would be of value to provide along with the results (others, please feel free to add more to the list for the results to be more useful)
- Architecture of the system

- base distro (e.g. /etc/release).

Quoted 1 line

- Configuration - memory size and number of procs

I think that all the info that we ask people to report in bug reports would be a good start....

Quoted 1 line

- Filesystem sizes

More useful is the MKFS_OPTIONS and MOUNT_OPTIONS used to run the tests, as that tells us how much non-default test coverage we are getting. i.e. default testing or something different.

Quoted 1 line

- Commit ID of the kernel

Not useful for kernels built with local, non-public changes, which is generally 100% of the kernels and userspace packages I test with.

Quoted 2 lines

- which git tree (XFS git tree or Linus's)
- xfsprogs version (or commit ID if from the git tree)

Same as for the kernel - base version is probably all that is useful here.

You'd probably also want to capture the console output indicating test runtimes and why certain tests weren't run.

If you create a pristine $RESULTS_DIR and output all the information you want to gather into it, then it will be trivial for users to send information onwards. Providing a command line parameter that generates a unique results directory and then packages the results up into a tarball would be a great start. We'd then have a single file that can be sent up a central point with all the test results available. We could even do all the filtering/processing before upload.

Quoted 6 lines

IOWs, the idea behind $RESULTS_DIR is to make this sort of scripted test result gathering simple to do....

Cheers,

Dave.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw

Hi Chandra,

Quoted 25 lines · Chandra Seetharaman

On Wed, 2013-06-19 at 09:34 +1000, Dave Chinner wrote:

Hi Dave,

Quoted 7 lines · Chandra Seetharaman
Quoted 6 lines

Hello All,

Couple of weeks backs we had a discussion in xfs meeting to collect xfstests results. I volunteered to collect xfstests results from different architectures and upload to XFS.org.

Quoted 6 lines
Quoted 3 lines

I can run and get the results for x86_64 and ppc64. If anyone has other architectures that they can run the tests on and provide me the results, I will filter them an upload to XFS.org.

How are you going to filter and display them on xfs.org? Should the scripts to do this be part of xfstests?

I wasn't thinking of very elaborate filtering of all the results submitted by all variants of kernel/xfsprogs.

My thinking was very simple:
- get test results based on publicly available git tree commit IDs
- show the commit ID, and the failures seen in a specific arch.

Quoted 5 lines

This would help when anybody runs xfstests and sees a failure with any newer code, they would know if it is a regression or already seen.

But, looks like there is more elaborate work in progress. I will sync up with Ben and Phil to see how to help.

Here is what little I have so far. One of my goals is to be able to track all of the testing in a centralized location for every patch that has been posted to the list. Tracking only the test results is useful, but I've found that it is not valuable as I'd like unless I have a pretty good idea what code was being tested. Message-ID seems like the best way to accomplish this mapping to mailing list archive.

Patch 1:

Grabs some information about the source trees in question. We don't want this to be xfs specific so it goes after an environment variable to figure out where to look for sources. Set SRCDIRS like a path:

export SRCDIRS="/path/to/kernel:/path/to/xfsprogs:/path/to/xfsdump:/path/to/xfstests"

It will print the git tree/branch/desc of each directory and any quilt patches you have applied. It prints md5sum/messageid/patchworkid for each patch that is applied. This doesn't address other workflows that use packaging, so I'm toying with the idea of adding the ability to specify a package name in here, which would allow those who prefer to use a packaging-based approach a way to include sources with their test results.

Here is some sample output, which I admit isn't very pretty:

# ./check -g auto
fatal: ref HEAD is not a symbolic ref
SRCDIRS       -- /root/xfs:/root/xfsprogs:/root/xfsdump:/root/xfstests
    /root/xfs:
        URL           -- git://oss.sgi.com/xfs/xfs.git
        BRANCH        -- refs/heads/master
        DESC          -- v3.10-rc1-39-g2fb8b50
        PATCHES:
                /root/xfs/patches/xfs-update-mount-options-documentation.patch
                md5sum:         09a25d03e65fd969c058d6dd39fc1315
                X-Patchwork-Id: 5621
                Message-Id:     <[email protected]>
                /root/xfs/patches/xfs-add-pluging-for-bulkstat-readahead.patch
                md5sum:         0c3772ba0792d42332139268e86036e2
                X-Patchwork-Id: 5620
                Message-Id:     <[email protected]>
...
                Message-Id:     <[email protected]>
                /root/xfs/patches/xfs-use-inode-create-transaction.patch
                md5sum:         fb7259b0b3eb7c29a3dfc3a487c1584e
                X-Patchwork-Id: 5630
                Message-Id:     <[email protected]>
    /root/xfsprogs:
        URL           -- git://oss.sgi.com/xfs/cmds/xfsprogs.git
        BRANCH        --
        DESC          -- v3.1.9
        PATCHES:
    /root/xfsdump:
        URL           -- git://oss.sgi.com/xfs/cmds/xfsdump.git
        BRANCH        -- refs/heads/3.1.2
        DESC          -- v3.1.2
    /root/xfstests:
        URL           -- git://oss.sgi.com/xfs/cmds/xfstests.git
        BRANCH        -- refs/heads/master
        DESC          -- linux-v3.8-131-ge2549c6
        PATCHES:
                /root/xfstests/patches/keep-track-10
                md5sum:         64f67d028b133bb540a29d2a1223492c
                X-Patchwork-Id:
                Message-Id:
                /root/xfstests/patches/xfstests-check-cleanup-check.log-update-4
                md5sum:         eb41327c478bf99adb62cd8db3258ad5
                X-Patchwork-Id:
                Message-Id:
                /root/xfstests/patches/putresults-4
                md5sum:         f41fb9bfe8b88e5bab2de440de91e47a
                X-Patchwork-Id:
                Message-Id:
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 xfsqa-sum1 3.10.0-rc1+
MKFS_OPTIONS  -- -f -bsize=4096 /dev/sdb1
MOUNT_OPTIONS -- /dev/sdb1 /mnt/scratch

generic/001 5s ... 5s

You can see I'm testing a chunk of the 3.11 queue on this machine... This gives us a way to map back from a set of test results to the exact message on the list.

Patch 2:

This guy just switches us to use a temporary check.log file so that it can be uploaded later. At the end it appends this to check.log in the results directory so that the behavior looks the same as before. Not much to see here.

Patch 3:

This one archives the description of what is being tested as well as those sources which are not in the git tree that are being tested. This should give you all the info you need to duplicate the test run at a later date. It also archives test results during the run, including timestamps of when each test starts and stops. This is done so that on the upload end a cron job will be able to tell whether a given system has hung in the middle of a test run, and take action if a test is taking longer than expected (future work). Currently it is using curl to upload via ftp, which you control by setting an environment variable:

export ARCHIVE_URL=ftp://anonymous:[email protected]/xfstests/results

I think it might be helpful to add a few more url types. Ideas that come to mind are dir://path/to/my/local/archive, and tar:/path/to/tarball.tar

Here's what the current directory structure looks like:

/ftp/xfstests/results/xfsqa-sum1.americas.sgi.com/3.10.0-rc1+/1371679906
hostname ^^^^^^^^^^^^^^^^^^^^^^^^^^^
kernel version ^^^^^^^^^^^
start date ^^^^^^^^^^

# ls
5015.start  5015.stop  check.desc  check.log  check.time  root/
# find root
root
root/xfs
root/xfs/patches
root/xfs/patches/xfs-update-mount-options-documentation.patch
...
root/xfs/patches/xfs-use-inode-create-transaction.patch
root/xfstests
root/xfstests/patches
root/xfstests/patches/keep-track-10
root/xfstests/patches/xfstests-check-cleanup-check.log-update-4
root/xfstests/patches/putresults-4

All of the patches are in there...

# cat *.st*
tests/generic/001 61919
tests/generic/001 61924

Those are the start and stop times for test 'generic/001'.

check.desc is the output you saw above, check.log is the standard output you'd expect but only for this run, check.time is the amount of time each test took, and you also get $seq.notrun, as well as test failure output uploaded.

-----

My future ideas are to write a perl script to parse through the test results and print some reports. Based upon that script, we could throw the output into a website or database, for people who are into that kind of thing.

So, in conclusion, there isn't a whole lot here! ;)

I would like to stress that it is important to keep track of what you're testing along with the results, that making the test archival function part of the test scripts themselves has some advantages, and hopefully this could be extended to address everyone's preferred workflow. It is important to have something that is useful for individual developers as well as commercial interests, without the fuss of a great deal of web/sql setup. I've made provisions for those who prefer to keep their testing private, and also for people who wish to mine their test results on a larger scale. Just set ARCHIVE_URL to something that suits your needs.

I'm hoping to explore some of those ideas again soon.

Thanks,
Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw

RFC PATCH 1/3 xfstests: get some basic source tree info

Grab some basic information about the souce trees being tested and save it off for later. This includes information about the git commit and any patches which are applied.

Set SRCDIRS environment variable with paths to the sources you're testing, colon delimited like PATH.

e.g.

export SRCDIRS="/path/to/kernel:/path/to/xfsprogs:/path/to/xfsdump:/path/to/xfstests"

---
 check     |    4 ++++
 common.rc |   36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)
check +4 -0
Index: xfstests/check
===================================================================
@@ -316,6 +316,7 @@ END	{ if (NR > 0) {
 
 	echo "" >>$check.log
 	date >>$check.log
+	cat /tmp/check.desc >>$check.log
 	echo $list | fmt | sed -e 's/^/    /' -e "s;$SRC_DIR/;;g" >>$check.log
 	$interrupt && echo "Interrupted!" >>$check.log
         
@@ -363,6 +364,9 @@ rm -f $check.full
 
 [ -f $check.time ] || touch $check.time
 
+_full_source_details > /tmp/check.desc
+cat /tmp/check.desc
+
 # print out our test configuration
 echo "FSTYP         -- `_full_fstyp_details`"
 echo "PLATFORM      -- `_full_platform_details`"
common/rc +42 -0

48 lines not on this page · raw message

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#7 lore raw

Re: [RFC PATCH 1/3] xfstests: get some basic source tree info

Quoted 2 lines · Ben Myers

Grab some basic information about the souce trees being tested and save it off for later.

save it where? Ah, results/check.log.

README updates maybe?

Quoted 9 lines

This includes information about the git commit and any patches which are applied.

Set SRCDIRS environment variable with paths to the sources you're testing, colon delimited like PATH.

e.g.

export SRCDIRS="/path/to/kernel:/path/to/xfsprogs:/path/to/xfsdump:/path/to/xfstests"

ok so after looking at the patch more carefully, this can be any collection of paths to git trees in any order, right, and outputs i.e.:

SRCDIRS -- /mnt/test2/git/linux-2.6:/mnt/test2/git/xfsdump:/mnt/test2/git/xfsprogs
/mnt/test2/git/linux-2.6:
URL -- git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
BRANCH -- refs/heads/master
DESC -- v3.10-rc2-48-g4284097
/mnt/test2/git/xfsdump:
URL -- git://oss.sgi.com/xfs/cmds/xfsdump.git
BRANCH -- refs/heads/master
DESC -- v3.1.2-4-g71f940f
...

But what if we're testing packaged bits, not from a git tree, should that be collected?

Quoted 51 lines
---
 check     |    4 ++++
 common.rc |   36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)

Index: xfstests/check
===================================================================

check +4 -0
@@ -316,6 +316,7 @@ END	{ if (NR > 0) {
 
 	echo "" >>$check.log
 	date >>$check.log
+	cat /tmp/check.desc >>$check.log
 	echo $list | fmt | sed -e 's/^/    /' -e "s;$SRC_DIR/;;g" >>$check.log
 	$interrupt && echo "Interrupted!" >>$check.log
 
@@ -363,6 +364,9 @@ rm -f $check.full
 
 [ -f $check.time ] || touch $check.time
 
+_full_source_details > /tmp/check.desc
+cat /tmp/check.desc
+
 # print out our test configuration
 echo "FSTYP         -- `_full_fstyp_details`"
 echo "PLATFORM      -- `_full_platform_details`"

Index: xfstests/common/rc
===================================================================

common/rc +14 -0
@@ -1738,6 +1738,48 @@ _full_platform_details()
      echo "$os/$platform $host $kernel"
 }
 
+_full_source_details()
+{
+    if [ -z $SRCDIRS ]; then
+        return
+    fi
+
+    echo "SRCDIRS       -- $SRCDIRS"
+
+    dirs=$(echo $SRCDIRS | tr ":" "\n")
+    for dir in $dirs
+    do
+        echo "    $dir:"
+        if [ -d $dir/.git ]; then
+            # git url, branch, and description

common/rc _full_platform_details() · line 1764

Quoted 10 lines
common/rc +10 -0
+            url=$(cd $dir; git remote show origin | grep 'Fetch URL' | awk '{print $3}')
+            echo -e "\tURL           -- $url"
+            branch=$(cd $dir; git symbolic-ref HEAD)
+            echo -e "\tBRANCH        -- $branch"
+            desc=$(cd $dir; git describe)
+            echo -e "\tDESC          -- $desc"
+        fi
+
+        if [ -d $dir/patches ]; then
+            # quilt patches which are applied

what if it's guilt not quilt?

Quoted 22 lines
common/rc +18 -0
+            echo -e "\tPATCHES:"
+            for p in $(cd $dir; quilt applied 2> /dev/null)
+            do
+                p=${p##patches/}
+                md5=$(md5sum $dir/patches/$p)
+                md5_checksum=${md5%% *}
+                patch_fn=${md5##* }
+                patchworkid=$(egrep -i '^X-Patchwork-Id:' $dir/patches/$p)
+                messageid=$(egrep -i '^Message-Id:' $dir/patches/$p)
+                echo -e "\t\t$patch_fn"
+                echo -e "\t\tmd5sum:\t\t$md5_checksum"
+                echo -e "\t\tX-Patchwork-Id:\t${patchworkid##* }"
+                echo -e "\t\tMessage-Id:\t${messageid##* }"
+            done
+        fi
+    done
+}
+
 _setup_udf_scratchdir()
 {
     [ "$FSTYP" != "udf" ] \
Quoted 5 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#8 lore raw

Re: [RFC PATCH 1/3] xfstests: get some basic source tree info

Quoted 32 lines · Eric Sandeen
Quoted 2 lines · Ben Myers

Grab some basic information about the souce trees being tested and save it off for later.

save it where? Ah, results/check.log.

README updates maybe?

Quoted 9 lines

This includes information about the git commit and any patches which are applied.

Set SRCDIRS environment variable with paths to the sources you're testing, colon delimited like PATH.

e.g.

export SRCDIRS="/path/to/kernel:/path/to/xfsprogs:/path/to/xfsdump:/path/to/xfstests"

ok so after looking at the patch more carefully, this can be any collection of paths to git trees in any order, right, and outputs i.e.:

SRCDIRS -- /mnt/test2/git/linux-2.6:/mnt/test2/git/xfsdump:/mnt/test2/git/xfsprogs
/mnt/test2/git/linux-2.6:
URL -- git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
BRANCH -- refs/heads/master
DESC -- v3.10-rc2-48-g4284097
/mnt/test2/git/xfsdump:
URL -- git://oss.sgi.com/xfs/cmds/xfsdump.git
BRANCH -- refs/heads/master
DESC -- v3.1.2-4-g71f940f
...

Quoted 2 lines

But what if we're testing packaged bits, not from a git tree, should that be collected?

Ideally, yes. My CI system builds packages, then ships them over to the VMs that run tests and installs them before the tests are run. So there's never any source trees available for such probing. And often I just use the versions install on the distro I'm doing my testing on.

Quoted 18 lines
Quoted 18 lines
common/rc +18 -0
+_full_source_details()
+{
+    if [ -z $SRCDIRS ]; then
+        return
+    fi
+
+    echo "SRCDIRS       -- $SRCDIRS"
+
+    dirs=$(echo $SRCDIRS | tr ":" "\n")
+    for dir in $dirs
+    do
+        echo "    $dir:"
+        if [ -d $dir/.git ]; then
+            # git url, branch, and description
+            url=$(cd $dir; git remote show origin | grep 'Fetch URL' | awk '{print $3}')
+            echo -e "\tURL           -- $url"
+            branch=$(cd $dir; git symbolic-ref HEAD)
+            echo -e "\tBRANCH        -- $branch"
Quoted 8 lines

common/rc _full_platform_details() · line 1764

Quoted 6 lines
common/rc +6 -0
+            desc=$(cd $dir; git describe)
+            echo -e "\tDESC          -- $desc"
+        fi
+
+        if [ -d $dir/patches ]; then
+            # quilt patches which are applied

what if it's guilt not quilt?

Then it will give a git signature that doesn't match anything that was ever published.

But from my perspective, that simply doesn't matter. I'm not going to be sending lists of patches I'm testing to a remote server - there's way too much scope for inadvertant information leaks in doing this. Especially considering patch 3/3 sends the *patches* to the remote server via the _archive function. There's no way I'd be recommending such functionality be used in environments that contain both upstream and internal development trees.

What you propose might be fine for individual developer usage, but it's not appropriate for public distribution of test results. Result logs for public distribution need to be as anonymous as possible with only the bare minimum of identifying information in them.

Hence for public consumption, I'd suggest that we'd do far better to just target released kernels and packages. e.g. the kernel release tag like 3.10-rc4, and the xfsprogs/dump/test version being used. Having a repository of full of failure reports from random developer patchsets and kernels is not useful to anyone....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#9 lore raw

Re: [RFC PATCH 1/3] xfstests: get some basic source tree info

Hey Eric & Dave,

Quoted 33 lines · Dave Chinner
Quoted 32 lines · Eric Sandeen
Quoted 2 lines · Ben Myers

Grab some basic information about the souce trees being tested and save it off for later.

save it where? Ah, results/check.log.

README updates maybe?

Quoted 9 lines

This includes information about the git commit and any patches which are applied.

Set SRCDIRS environment variable with paths to the sources you're testing, colon delimited like PATH.

e.g.

export SRCDIRS="/path/to/kernel:/path/to/xfsprogs:/path/to/xfsdump:/path/to/xfstests"

ok so after looking at the patch more carefully, this can be any collection of paths to git trees in any order, right, and outputs i.e.:

SRCDIRS -- /mnt/test2/git/linux-2.6:/mnt/test2/git/xfsdump:/mnt/test2/git/xfsprogs
/mnt/test2/git/linux-2.6:
URL -- git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
BRANCH -- refs/heads/master
DESC -- v3.10-rc2-48-g4284097
/mnt/test2/git/xfsdump:
URL -- git://oss.sgi.com/xfs/cmds/xfsdump.git
BRANCH -- refs/heads/master
DESC -- v3.1.2-4-g71f940f
...

Quoted 2 lines
Quoted 2 lines

But what if we're testing packaged bits, not from a git tree, should that be collected?

IMO we should support both git tree and packaged workflows.

Quoted 5 lines

Ideally, yes. My CI system builds packages, then ships them over to the VMs that run tests and installs them before the tests are run. So there's never any source trees available for such probing. And often I just use the versions install on the distro I'm doing my testing on.

Any suggestions on how to track the sources the packaged workflow? My first thought is that we could package sources in the .rpm or .deb or srpm and ship that over to the test machine. Then add the name of the rpm/deb to the SRCDIRS envvar and teach this thing to grab the sources from there. But if you're talking about a kernel-source rpm this could get fairly large.

Alternatively, if you have a way of tagging what you're testing and linking that up with your SCM you don't have to go after the sources on the test box, just make sure to make a note of the tag. Maybe add the git url/branch/desc the rpm info which you could query with -qi?

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
common/rc +18 -0
+_full_source_details()
+{
+    if [ -z $SRCDIRS ]; then
+        return
+    fi
+
+    echo "SRCDIRS       -- $SRCDIRS"
+
+    dirs=$(echo $SRCDIRS | tr ":" "\n")
+    for dir in $dirs
+    do
+        echo "    $dir:"
+        if [ -d $dir/.git ]; then
+            # git url, branch, and description
+            url=$(cd $dir; git remote show origin | grep 'Fetch URL' | awk '{print $3}')
+            echo -e "\tURL           -- $url"
+            branch=$(cd $dir; git symbolic-ref HEAD)
+            echo -e "\tBRANCH        -- $branch"
Quoted 8 lines
Quoted 8 lines

common/rc _full_platform_details() · line 1764

Quoted 6 lines
common/rc +6 -0
+            desc=$(cd $dir; git describe)
+            echo -e "\tDESC          -- $desc"
+        fi
+
+        if [ -d $dir/patches ]; then
+            # quilt patches which are applied

what if it's guilt not quilt?

Right now I'm a quilt guy... but I think it would be nice if we could support both.

Quoted 11 lines

Then it will give a git signature that doesn't match anything that was ever published.

But from my perspective, that simply doesn't matter.

I'm not going to be sending lists of patches I'm testing to a remote server - there's way too much scope for inadvertant information leaks in doing this. Especially considering patch 3/3 sends the *patches* to the remote server via the _archive function. There's no way I'd be recommending such functionality be used in environments that contain both upstream and internal development trees.

I want to do this in a way that doesn't force users to use it. Maybe add an --upload option to check so that it's clearer whether you're going to upload?

Quoted 2 lines

What you propose might be fine for individual developer usage, but it's not appropriate for public distribution of test results.

I'm looking for something that is flexable enough for both individual developer usage as well as public distribution, in the same mechanism.

Quoted 3 lines

Result logs for public distribution need to be as anonymous as possible with only the bare minimum of identifying information in them.

To what end? We're already posting to a publicly available mailing list which is archived for posterity. There is no anonymity here. I guess we could hash the hostname into something unrecognisable? It's worth keeping an email address, I think.

Quoted 3 lines

Hence for public consumption, I'd suggest that we'd do far better to just target released kernels and packages. e.g. the kernel release tag like 3.10-rc4, and the xfsprogs/dump/test version being used.

I tend to agree that for public consumption you'd want to upload only tthe results of tagged versions of the kernels/xfsprogs/dump/test. You generally wouldn't want to upload results from day-to-day development work to a public location, but this does allow you to upload those results to an internal server if you want to.

However, if Jane Developer wants to post test results for the patches she has published, I want to enable her to do so... maybe to a different location than the tagged versions.

Quoted 2 lines

Having a repository of full of failure reports from random developer patchsets and kernels is not useful to anyone....

Too much data from test reports is a problem I'd love to have. ;)

-Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#10 lore raw

RFC PATCH 2/3 xfstests: use an intermediate check.log file

In reply to Ben Myers

Use an intermediate check.log file for each test run so that it can be uploaded to a central location by a subsequent patch in this series. The intermediate file is appended to $RESULTS_DIR/check.log a the end of each run.

---
 check |   21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)
check +11 -9
Index: xfstests.new2/check
===================================================================
@@ -314,11 +314,11 @@ END	{ if (NR > 0) {
 	    mv $tmp.out $check.time
 	fi
 
-	echo "" >>$check.log
-	date >>$check.log
-	cat /tmp/check.desc >>$check.log
-	echo $list | fmt | sed -e 's/^/    /' -e "s;$SRC_DIR/;;g" >>$check.log
-	$interrupt && echo "Interrupted!" >>$check.log
+	echo "" >>/tmp/check.log
+	date >>/tmp/check.log
+	cat /tmp/check.desc >>/tmp/check.log
+	echo $list | fmt | sed -e 's/^/    /' -e "s;$SRC_DIR/;;g" >>/tmp/check.log
+	$interrupt && echo "Interrupted!" >>/tmp/check.log
         
         if [ ! -z "$n_try" -a $n_try != 0 ]
 	then
@@ -328,20 +328,22 @@ END	{ if (NR > 0) {
 	if [ ! -z "$notrun" ]
 	then
 	    echo "Not run:$notrun"
-	    echo "Not run:$notrun" >>$check.log
+	    echo "Not run:$notrun" >>/tmp/check.log
 	fi
 
         if [ ! -z "$n_bad" -a $n_bad != 0 ]
 	then
 	    echo "Failures:$bad"
 	    echo "Failed $n_bad of $n_try tests"
-	    echo "Failures:$bad" | fmt >>$check.log
-	    echo "Failed $n_bad of $n_try tests" >>$check.log
+	    echo "Failures:$bad" | fmt >>/tmp/check.log
+	    echo "Failed $n_bad of $n_try tests" >>/tmp/check.log
 	else
 	    echo "Passed all $n_try tests"
-	    echo "Passed all $n_try tests" >>$check.log
+	    echo "Passed all $n_try tests" >>/tmp/check.log
 	fi
 	needwrap=false
+        cat /tmp/check.log >> $check.log
+	rm -f /tmp/check.log
     fi
 
     rm -f /tmp/*.rawout /tmp/*.out /tmp/*.err /tmp/*.time

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#11 lore raw

RFC PATCH 3/3 xfstests: upload test results

In reply to Ben Myers

Set the ARCHIVE_URL environment variable to upload test results to a central location:

export ARCHIVE_URL=ftp://anonymous:[email protected]/xfstests/results

---
 check     |   14 ++++++++++++++
 common.rc |   24 ++++++++++++++++++++++++
 2 files changed, 38 insertions(+)
check +13 -1
Index: xfstests/check
===================================================================
@@ -28,6 +28,7 @@ n_bad=0
 bad=""
 notrun=""
 interrupt=true
+start_time=`date "+%s"`
 diff="diff -u"
 showme=false
 have_test_arg=false
@@ -312,6 +313,7 @@ END	{ if (NR > 0) {
 	}' \
 	    | sort -n >$tmp.out
 	    mv $tmp.out $check.time
+	    _archive $check.time $start_time
 	fi
 
 	echo "" >>/tmp/check.log
@@ -343,6 +345,7 @@ END	{ if (NR > 0) {
 	fi
 	needwrap=false
         cat /tmp/check.log >> $check.log
+	_archive /tmp/check.log $start_time
 	rm -f /tmp/check.log
     fi
 
@@ -368,6 +371,7 @@ rm -f $check.full
 
 _full_source_details > /tmp/check.desc
 cat /tmp/check.desc
+_archive /tmp/check.desc $start_time
 
 # print out our test configuration
 echo "FSTYP         -- `_full_fstyp_details`"
@@ -453,6 +457,8 @@ do
 	rm -f core $seqres.notrun
 
 	start=`_wallclock`
+	echo "$seq $start" > $tmp.start
+	_archive $tmp.start $start_time
 	$timestamp && echo -n "	["`date "+%T"`"]"
 	[ ! -x $seq ] && chmod u+x $seq # ensure we can run it
 	$LOGGER_PROG "run xfstest $seqnum"
@@ -460,6 +466,8 @@ do
 	sts=$?
 	$timestamp && _timestamp
 	stop=`_wallclock`
+	echo "$seq $stop" > $tmp.stop
+	_archive $tmp.stop $start_time
 
 	_fix_malloc <$tmp.rawout >$tmp.out
 	rm -f $tmp.rawout
@@ -468,6 +476,7 @@ do
 	then
 	    echo -n " [dumped core]"
 	    mv core $RESULT_BASE/$seqnum.core
+	    _archive $RESULT_BASE/$seqnum.core $start_time
 	    err=true
 	fi
 
@@ -476,6 +485,7 @@ do
 	    $timestamp || echo -n " [not run] "
 	    $timestamp && echo " [not run]" && echo -n "	$seqnum -- "
 	    cat $seqres.notrun
+            _archive $seqres.notrun $start_time
 	    notrun="$notrun $seqnum"
 	else
 	    if [ $sts -ne 0 ]
@@ -501,6 +511,8 @@ do
 		else
 		    echo " - output mismatch (see $seqres.out.bad)"
 		    mv $tmp.out $seqres.out.bad
+		    _archive $seqres.out $start_time
+		    _archive $seqres.out.bad $start_time
 		    $diff $seq.out $seqres.out.bad | {
 		        if test "$DIFF_LENGTH" -le 0; then
 				cat
@@ -515,7 +527,7 @@ do
 		fi
 	    fi
 	fi
-
+        rm -f $tmp.$seq.start $tmp.seq.stop
     fi
 
     # come here for each test, except when $showme is true
common/rc +22 -0
Index: xfstests/common/rc
===================================================================
@@ -1738,6 +1738,27 @@ _full_platform_details()
      echo "$os/$platform $host $kernel"
 }
 
+_archive()
+{
+    if [ -z "$ARCHIVE_URL" ]; then
+        return;
+    fi
+
+    file=$1
+    subdir=$2
+    args=$3
+
+    if [ ! -e $file ]; then
+        _fail "archive: $file does not exist"
+    fi
+
+    desc=$(cat /proc/version | awk '{print $3}')
+
+    url=$ARCHIVE_URL/`hostname -f`/$desc/$subdir
+    curl $args --ftp-create-dirs --silent --upload-file \
+         $file $url/`basename $file`
+}
+
 _full_source_details()
 {
     if [ -z $SRCDIRS ]; then
@@ -1775,6 +1796,7 @@ _full_source_details()
                 echo -e "\t\tmd5sum:\t\t$md5_checksum"
                 echo -e "\t\tX-Patchwork-Id:\t${patchworkid##* }"
                 echo -e "\t\tMessage-Id:\t${messageid##* }"
+                _archive $dir/patches/$p $start_time/$dir/patches
             done
         fi
     done

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs