Add u-boot command regression tests.

This uses PyUnit and python-serial <http://pyserial.sourceforge.net/>
to do unit testing on u-boot commands.

---

...and now, for something completely different.

The concept here is to use PyUnit testing in conjunction with python serial I/O (theoretically, ethernet-based command I/O could be handled too).

This is a concept probably sufficiently implemented for the "help" command and a good start for the "fdt" command.

What does The List think?
* Useful to have regression tests for the u-boot commands?
* Implementation-wise, am I heading in the right direction?

Best regards,
gvb

 u-unit/.gitignore |    2 +
 u-unit/fdt.dts    |   30 +++++++++
 u-unit/fdt.py     |  171 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 u-unit/help.py    |   70 ++++++++++++++++++++++
 u-unit/ubootio.py |   55 +++++++++++++++++
 5 files changed, 328 insertions(+), 0 deletions(-)
 create mode 100644 u-unit/.gitignore
 create mode 100644 u-unit/fdt.dts
 create mode 100755 u-unit/fdt.py
 create mode 100755 u-unit/help.py
 create mode 100755 u-unit/ubootio.py

new file mode 100644
index 0000000..88e6bec
@@ -0,0 +1,2 @@
+*.pyc
+*.dtb

new file mode 100644
index 0000000..66efcdd
@@ -0,0 +1,30 @@
+/dts-v1/;
+
+/ {
+	compatible = "fdt";
+	prop-int = <0xdeadbeef>;
+	prop-str = "hello world";
+
+	subnode at 1 {
+		compatible = "subnode1";
+		prop-int = <0x01234567>;
+
+		subsubnode {
+			compatible = "subsubnode1", "subsubnode";
+			prop-int = <0x12345678>;
+			prop-empty;
+		};
+	};
+
+	subnode at 2 {
+		linux,phandle = <0x2000>;
+		prop-int = <0x23456789>;
+
+		subsubnode at 0 {
+			linux,phandle = <0x2001>;
+			compatible = "subsubnode2", "subsubnode";
+			prop-empty;
+			prop-int = <0x3456789a>;
+		};
+	};
+};

1.5.3.4

Hello,

It is not only useful, but IMHO absolutely necessary to do regular regression testing on U-Boot.

I wish you had asked that question before spending efforts on this. The point is, that there is already a pretty extensive and extandable regression test suite for U-Boot (and Linux, and some more). We just failed to announce it loudly enough (which clearly shows that DENX is not marketing driven - and funny enough, I'm not a bit ashamed about that). However, I'm sad about the avoidable duplication of efforts, so I have to say I'm sorry.

Please have a look at the DUTS, our regression test suite. See the git repo at http://www.denx.de/cgi-bin/gitweb.cgi?p=duts.git;a=summary
and some documentation at http://www.denx.de/wiki/DUTS/DUTSDocs

I'd really appreciate if this was picked up by others as well. [It took us a long and winded road to get there - trying some existing solutions like dejagnu etc. and finally ending with a do-it-yourself solution.]

Note: there is a non-abvious benefit from the DUTS: once you got it running on a board, it will generate a set of log files which just need to be uploaded to our web server to generate a new version of the DULG for this board. The idea is that if you have a new software version you just need to run the DUTS to get (1) a confirmation that everything is working as expected fromt he regression test suite and
(2) an updated version of the documentation with eexamples showing exacly this software version on exactly that hardware.

Best regards,

Wolfgang Denk

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Es gibt immer genug fuer die Beduerfnisse aller, aber  niemals  genug
fuer die Gier einzelner.                                    -- Ghandi

Oh no apology necessary. It was a learning exercise, useful in its own right, WRT Python, PyUnit, PySerial, and seeing what I could do with them.

I've used tk/tcl/expect and am not wild about them. The older scripting languages are so.... 70s. ;-) I haven't used python much, but I've come to like it a lot, kind of a rational perl. :-/

My experience with expect is that it worked well for simple cases, but when I started matching more patterns and more complex patterns, it would get harder and harder to match reliably. Also, its asynchronous timing-based capture (which causes "capture breaks" in different places when you least expect it) drove me crazy. Then to debug the pattern matching (or not matching, or mismatching, as the case usually was) was difficult.

Having said that, that was a few years ago and maybe I'm smarter now...

Thanks for the pointer. I've poked around a bit and will continue to do so.

Best regards,
gvb

Hi,

I think I know what you mean.

I'm open to all suggestions for improvement, but there is a certain complexity in the setup itself, and I cannot say how much can be avoided by using other tools.

Best regards,

Wolfgang Denk

DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
If you believe that feeling bad or worrying long enough will change a
past or future event, then you are residing on another planet with  a
different reality system.

Same here, we are through all these iterations as well:

- we've written a test suite in python with xml specs (oo design, but very complex in the end, without the advantages we expected from an oo language)

- expect/tcl based test suite; causes brain cancer because of tcl

Now we do the u-boot tests/scripting with the tcl/expect stuff and all other Linux test suites in normal shell scripts, remote-controlled via ssh.

While the latter one has finally turned out to be a good way to write test cases, we think about adding something like a dbus module to u-boot-v2, which would make it possible to do real and well defined RPC between a host and u-boot.

rsc

 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Hmmm, I seem to have hit a resonance.

More background trivia... my weekend experiment was using python-serial.
I chose that and used it to capture whole serial transactions (waited
until it appeared no more serial characters were being sent) so I could then use python regexp matching to do simpler, more elaborate, and more sane (yes, all three) pattern matching than what I was able to do with expect due to expect breaking the serial stream in semi-random time based chunks.

Another option that I have not experimented with is a python-based expect:
<http://sourceforge.net/projects/pexpect/>

Best regards,
gvb

:-)

That was what we used; two python sub-versions later they had changed some internal interfaces and everything stopped working. Nothing that looks like a stable platform to depend on ...

Robert

 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

I assume you'll be dropping this patch from your current libfdt tree in the testing branch.

- k

Yes. (In retrospect, I should have made a separate branch as well.) I just dropped the spurious patch and re-pushed the testing branch (I kept the trivial "spaces around the =" print format change that I added just before the regression testing patch).

Thanks for reminding me to clean up,
gvb

i think that's really nature of the beast and not a deficiency in expect. how do you know when things stop coming in ? you dont ... so write properly anchored input matches and things work nicely
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 827 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20071128/8dda484f/attachment.pgp