Skip to contentSkip to conversation

livetime of boards

U-Boot Heiko Schocher 5 participants 31 messages Nov 5, 2013 – Nov 8, 2013

lore
Share & tools
lore
#0 lore raw

Hello all,

We have the real problem, that we have a lot of old boards, which are unmaintained in U-Boot, and we have no chance to find out, if this boards are longer used/tested ...

So, the question raises, should we introduce a column in boards.cfg, which shows the "livetime" of a board support in U-Boot?

If we introduce this, we init the livetime column in boards.cfg with a value of n for all current boards, or if we add a new board support. (n releases valid) I think n = 4 would be a good starting point.

Every release cycle the livetime value gets decremented by one. (Can be done by a script sitting in the tools directory?)

livetime == 0 -> EMail to board maintainer, that board livetime ends.
please test and send an update, if board support should remain in U-Boot.

livetime == -1 -> board gets deleted in next release

If a board maintainer gets the above EMail, or of course whenever he really try a current release on a real hardware, he can send a patch, which updates the livetime column again back to n ...

So, the hope is, we have after n releases only real tested and used boards in U-Boot...

What do others think?

short ToDo list:
- make an initial patch for boards.cfg
- make a script, which decrements the "livetime" column in boards.cfg and send an EMail for the "livetime == 0" case to the boardmaintainer and make a delete patch for boards.cfg if "livetime == -1" ?
- add a doc/README.livetime

bye,
heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#1 lore raw
Quoted 5 lines · Heiko Schocher

Hello all,

We have the real problem, that we have a lot of old boards, which are unmaintained in U-Boot, and we have no chance to find out, if this boards are longer used/tested ...

We also have a feature, lots of hardware support because lots of things don't change drastically, frequently. That's not to say that I wouldn't mind dropping old platforms (even that ones I have sentimental feelings towards), and would certainly like to see more and frequent sanity testing.

This problem comes up when we talk about doing big changes, and I think that's the right time to talk about things. And I think the answer should be, we try and convert things forward and when it's not obvious if things will still work correctly, or how to do it, that's when we need to make a hard push on the board maintainers to find some time to work on things.

It's not that being a U-Boot board maintainer is a near zero effort, when someone steps up to own a board, but to be frank, it also shouldn't be a frequent high-touch area either.

Quoted 2 lines

So, the question raises, should we introduce a column in boards.cfg, which shows the "livetime" of a board support in U-Boot?

I sense a lot of conflicting patches.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20131105/b05cdae8/attachment.pgp>
#2 lore raw

Dear Tom,

Quoted 10 lines · In message <20131105203736.GM5925@bill-the-cat> you wrote:
Quoted 4 lines

We have the real problem, that we have a lot of old boards, which are unmaintained in U-Boot, and we have no chance to find out, if this boards are longer used/tested ...

We also have a feature, lots of hardware support because lots of things don't change drastically, frequently. That's not to say that I wouldn't mind dropping old platforms (even that ones I have sentimental feelings towards), and would certainly like to see more and frequent sanity testing.

I think Heiko's idea of documenting test reports is pretty cool - but of course we need to discuss in detail how to implement this, and also decide wether we use this for (semi-automatic) code cleanup (by removing boards that have not been tested for a long time).

Quoted 6 lines

This problem comes up when we talk about doing big changes, and I think that's the right time to talk about things. And I think the answer should be, we try and convert things forward and when it's not obvious if things will still work correctly, or how to do it, that's when we need to make a hard push on the board maintainers to find some time to work on things.

Agreed. And here information how recently (or maybe even how frequently) a board has been tested (build tested, run on actual hardware) would come in really handy. we can probbaly automate build testing one way or another, but for actual runtime tests we will lways depend on the board maintainers, or board users.

Quoted 4 lines
Quoted 2 lines

So, the question raises, should we introduce a column in boards.cfg, which shows the "livetime" of a board support in U-Boot?

I sense a lot of conflicting patches.

Again I agree. Also, I fear that boards.cfg is becoming more and more unreadable by adding even more stuff. If I see this correctly, the maximum line length in boards.cfg already exceeds 360 characters :-(

So nstead of adding this information to boards.cfg we could probably use separate files for such information. We could provide tools to make test reports really easy, say something like

scripts/build_test
scripts/run_test

which the user would just call with a "passed" or "failed" argument; the scripts could then auto-detect which configuration and which exact U-Boot version were in use, and send an email. Whether that would be a patch against the source code or something that get's auto-added to a wiki page is just an implementation detail. But if we had something like this, we could get a muchbetter understanding how actively boards are being tested.

So when you're once again doing some change that requires touching files for some othe rboards, you could simply check that database. If you see that 3 out of the last 5 releases have reported succesful run-time tests you will probably decide to accept the needed efforts, but when you see the last test report is more than 5 years old, you will probably rather decide to initiate a code removal process.

Best regards,

Wolfgang Denk

Signature · 6 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
A Chairman was as necessary to a Board planet  as  the  zero  was  in
mathematics, but being a zero had big disadvantages...
                         - Terry Pratchett, _The Dark Side of the Sun_
#3 lore raw

Hello Wolfgang, Tom,

Am 06.11.2013 08:50, schrieb Wolfgang Denk:

Quoted 13 lines

Dear Tom,

Quoted 10 lines · In message<20131105203736.GM5925@bill-the-cat> you wrote:
Quoted 4 lines

We have the real problem, that we have a lot of old boards, which are unmaintained in U-Boot, and we have no chance to find out, if this boards are longer used/tested ...

We also have a feature, lots of hardware support because lots of things don't change drastically, frequently. That's not to say that I wouldn't mind dropping old platforms (even that ones I have sentimental feelings towards), and would certainly like to see more and frequent sanity testing.

Yep, thats the reason I had in mind. We must have more real tests on real hardware (or at least, get back the info, that this is done somewhere ...) ... thats the idea behind to force the board maintainers to give back us such info.

Quoted 4 lines

I think Heiko's idea of documenting test reports is pretty cool - but of course we need to discuss in detail how to implement this, and also decide wether we use this for (semi-automatic) code cleanup (by removing boards that have not been tested for a long time).

I vote for removing boards from which we get no such test info back. The board support code is just availiable, just not for current releases.

If code does not change drastically, such a "compile and try it on the hardware, give feedback to mainline" should not cost much time the board maintainer, and maybe we add a script, which the board maintainer just have to call, to send an "board tested EMail" to the mailinglist ...

Maybe a lot of boardmaintainers do this tests, and this info is important for mainline, but we have no mechanism to collect this info!

And if code changes drastically (which it currently does and will do in future I think) a board maintainer must decide, if it makes sense to sync the board with current mainline or the board get dropped from mainline ...

And I have this problem currently with i2c subsystem ... a lot of boards to convert to new framework, and I cannot decide, which are really maintained... or if currently, the converted boards, really work. (And I think, not only the i2c subsystem has this problem)

Quoted 7 lines
Quoted 6 lines

This problem comes up when we talk about doing big changes, and I think that's the right time to talk about things. And I think the answer should be, we try and convert things forward and when it's not obvious if things will still work correctly, or how to do it, that's when we need to make a hard push on the board maintainers to find some time to work on things.

Quoted 5 lines

Agreed. And here information how recently (or maybe even how frequently) a board has been tested (build tested, run on actual hardware) would come in really handy. we can probbaly automate build testing one way or another, but for actual runtime tests we will lways depend on the board maintainers, or board users.

Hmm.. Is it always obvious, if changes are big? Do we really get feedback when doing such big changes? I just reworking the i2c framework, and I have not really much feedback from board maintainers for their boards I changed ... Does this change really work on all boards I converted? I have no chance to get this info. But if we have such a livetime feature, I can be sure, that after n releases all boards in mainline are tested ... automatically ... I want such a feature ...

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

So, the question raises, should we introduce a column in boards.cfg, which shows the "livetime" of a board support in U-Boot?

I sense a lot of conflicting patches.

Again I agree. Also, I fear that boards.cfg is becoming more and more unreadable by adding even more stuff. If I see this correctly, the maximum line length in boards.cfg already exceeds 360 characters :-(

Right, boards.cfg gets unhandy ... Hmm .. what with the column "Staus" ... instead of "Active" it would be more informative to have there the livetime counter, and a single digit saves some characters ;-)

But you are right, that approach leads in a lot of conflicting patches ... but I think, we just pooled board information in boards.cfg, so this would be the right place in my eyes ...

Maybe we get such Information "a Boards is tested with current mainline" inform of an EMail with an Text "Board xy tested with commit mm. Please update livetime" ... and we can add a script, which updates the livetime for this board, so we can prevent conflicting patches ... ?

Quoted 7 lines

So nstead of adding this information to boards.cfg we could probably use separate files for such information. We could provide tools to make test reports really easy, say something like

scripts/build_test
scripts/run_test

Quoted 7 lines

which the user would just call with a "passed" or "failed" argument; the scripts could then auto-detect which configuration and which exact U-Boot version were in use, and send an email. Whether that would be a patch against the source code or something that get's auto-added to a wiki page is just an implementation detail. But if we had something like this, we could get a muchbetter understanding how actively boards are being tested.

Yes, that sound also good. I want to see the test information in the sourcecode, not somewhere on a wiki...

Quoted 4 lines

So when you're once again doing some change that requires touching files for some othe rboards, you could simply check that database. If you see that 3 out of the last 5 releases have reported succesful run-time tests you will probably decide to accept the needed efforts,

Hmm.. that works, if you have to touch some (some < 5) boards. But If you have to touch > 5 boards, this gets unhandy...

Quoted 2 lines

but when you see the last test report is more than 5 years old, you will probably rather decide to initiate a code removal process.

If we decide to delete older boards after n release cycles without testreports, we must not decide nor look in a database. We are sure, we have only "good and working" boards ... and we just do the necessary work for new features ... and we are sure, that we get back testreports within n release cycles ...

So let us decide first, if we want to go this way ...

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#4 lore raw

Hello all together,

Quoted 2 lines · Heiko Schocher

Am 06.11.2013 08:50, schrieb Wolfgang Denk:

Quoted 1 line

In message<20131105203736.GM5925@bill-the-cat> you wrote:

<snip problem description>

Full ACK, we need some way to track which board is working with the current ToT or at least on a release basis.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

So, the question raises, should we introduce a column in boards.cfg, which shows the "livetime" of a board support in U-Boot?

I sense a lot of conflicting patches.

Quoted 8 lines
Quoted 4 lines

Again I agree. Also, I fear that boards.cfg is becoming more and more unreadable by adding even more stuff. If I see this correctly, the maximum line length in boards.cfg already exceeds 360 characters :-(

Right, boards.cfg gets unhandy ... Hmm .. what with the column "Staus" ... instead of "Active" it would be more informative to have there the livetime counter, and a single digit saves some characters ;-)

I can't understand the status field at all, just for the record ;)

Quoted 8 lines

But you are right, that approach leads in a lot of conflicting patches ... but I think, we just pooled board information in boards.cfg, so this would be the right place in my eyes ...

Maybe we get such Information "a Boards is tested with current mainline" inform of an EMail with an Text "Board xy tested with commit mm. Please update livetime" ... and we can add a script, which updates the livetime for this board, so we can prevent conflicting patches ... ?

I agree here with Tom. Beside the possibility of conflicting pahces I see another problem here.
We will get a lot of patches just for increasing the tested counter for a single board. These patches needs to be handled in some way. If we shift to some integrated system (gerrit comes to mind) this could be easier than today, but it will bind resources anyways. Therefore I think it is a bad idea to save a such often changing information in the source code repository.

Quoted 15 lines
Quoted 7 lines

So nstead of adding this information to boards.cfg we could probably use separate files for such information. We could provide tools to make test reports really easy, say something like

scripts/build_test
scripts/run_test

Quoted 7 lines

which the user would just call with a "passed" or "failed" argument; the scripts could then auto-detect which configuration and which exact U-Boot version were in use, and send an email. Whether that would be a patch against the source code or something that get's auto-added to a wiki page is just an implementation detail. But if we had something like this, we could get a muchbetter understanding how actively boards are being tested.

Quoted 2 lines

Yes, that sound also good. I want to see the test information in the sourcecode, not somewhere on a wiki...

I think another place than the source code repository would be best for gathering such frequently changing information. Why not use some wiki other other web service for that purpose?

I don't want to search a web page for the information 'board X is not tested since ...' either. But we could easily write some scripts and add them to the source code repository to provide it.

Quoted 7 lines
Quoted 4 lines

So when you're once again doing some change that requires touching files for some othe rboards, you could simply check that database. If you see that 3 out of the last 5 releases have reported succesful run-time tests you will probably decide to accept the needed efforts,

Hmm.. that works, if you have to touch some (some < 5) boards. But If you have to touch > 5 boards, this gets unhandy...

How about:

MAKEALL --check-boards -s at91

;)

Quoted 2 lines
Quoted 2 lines

but when you see the last test report is more than 5 years old, you will probably rather decide to initiate a code removal process.

Why not save the SHA1 with the build-/runtime-tested information? Then we could easily build helper scripts to query that database when this board was last tested.

Quoted 7 lines

If we decide to delete older boards after n release cycles without testreports, we must not decide nor look in a database. We are sure, we have only "good and working" boards ... and we just do the necessary work for new features ... and we are sure, that we get back testreports within n release cycles ...

So let us decide first, if we want to go this way ...

Yes, we should introduce some mechanism to check when a specific board was last runtime tested. But I fear the overhead with patches that update a tested counter.

Best regards

Andreas Bie?mann

author Tested-by ...
#5 lore raw

Hello Andreas,

Am 07.11.2013 10:37, schrieb Andreas Bie?mann:

Quoted 21 lines

Hello all together,

Quoted 2 lines · Heiko Schocher

Am 06.11.2013 08:50, schrieb Wolfgang Denk:

Quoted 1 line

In message<20131105203736.GM5925@bill-the-cat> you wrote:

<snip problem description>

Full ACK, we need some way to track which board is working with the current ToT or at least on a release basis.

Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

So, the question raises, should we introduce a column in boards.cfg, which shows the "livetime" of a board support in U-Boot?

I sense a lot of conflicting patches.

Again I agree. Also, I fear that boards.cfg is becoming more and more unreadable by adding even more stuff. If I see this correctly, the maximum line length in boards.cfg already exceeds 360 characters :-(

Quoted 5 lines
Quoted 3 lines

Right, boards.cfg gets unhandy ... Hmm .. what with the column "Staus" ... instead of "Active" it would be more informative to have there the livetime counter, and a single digit saves some characters ;-)

I can't understand the status field at all, just for the record ;)

Hmm.. good question ...

Quoted 9 lines
Quoted 8 lines

But you are right, that approach leads in a lot of conflicting patches ... but I think, we just pooled board information in boards.cfg, so this would be the right place in my eyes ...

Maybe we get such Information "a Boards is tested with current mainline" inform of an EMail with an Text "Board xy tested with commit mm. Please update livetime" ... and we can add a script, which updates the livetime for this board, so we can prevent conflicting patches ... ?

Quoted 8 lines

I agree here with Tom. Beside the possibility of conflicting pahces I see another problem here.
We will get a lot of patches just for increasing the tested counter for a single board. These patches needs to be handled in some way. If we shift to some integrated system (gerrit comes to mind) this could be easier than today, but it will bind resources anyways. Therefore I think it is a bad idea to save a such often changing information in the source code repository.

I see this info just changing once when releasing a new U-Boot version.

I not see anymore a patch for updating the livetime counter for every board, instead we should have a script which a board maintainer can call, after he did a board test, which sends an EMail to the U-Boot ML with a special format, saying:
------------------------------------
subject: livetime: board name

Tested-by ...

with commit ...
------------------------------------
On the mailserver a script scans all incoming EMails for his Subject, (Is this possible?)
and collect the infos, for which boards, such EMails arrived. When releasing a new U-Boot version, this collected info can be used to update this livetime counter through another script saying "collect_livetime_info" (also this script can automatically send EMails to board maintainers, for boards which had reached end of livetime, outputs a delete board lists ...)

So (in current case Tom) should, before releasing a new U-Boot version, first call this script "collect_livetime_info" and he get:

-> one livetime counter patch for current release
-> one list for boards which reach end of life
-> one list for boards, which should be deleted

All Infos are "release info" I think, and fully fit in the commit for the new release ...

... maybe "deleting boards" can be done automatically, but this is not a trivial job ...

So, with such a solution, I see no big additional cost for adding such a feature (except the task "deleting old boards", which is maybe not trivial)

Do not understand me wrong, if we find another solution, I am happy also ... just spinning around ...

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines

So nstead of adding this information to boards.cfg we could probably use separate files for such information. We could provide tools to make test reports really easy, say something like

scripts/build_test
scripts/run_test

which the user would just call with a "passed" or "failed" argument; the scripts could then auto-detect which configuration and which exact U-Boot version were in use, and send an email. Whether that would be a patch against the source code or something that get's auto-added to a wiki page is just an implementation detail. But if we had something like this, we could get a muchbetter understanding how actively boards are being tested.

Quoted 6 lines
Quoted 2 lines

Yes, that sound also good. I want to see the test information in the sourcecode, not somewhere on a wiki...

I think another place than the source code repository would be best for gathering such frequently changing information. Why not use some wiki other other web service for that purpose?

See above explanation, I see this info not frequently changing, just always with a new U-Boot release ... and can nearly (except the "delete old boards" case) automatised ...

Quoted 3 lines

I don't want to search a web page for the information 'board X is not tested since ...' either. But we could easily write some scripts and add them to the source code repository to provide it.

Ok, fine with me too. I just thinking about this problem, and how we can fix it ;-)

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

So when you're once again doing some change that requires touching files for some othe rboards, you could simply check that database. If you see that 3 out of the last 5 releases have reported succesful run-time tests you will probably decide to accept the needed efforts,

Quoted 8 lines
Quoted 2 lines

Hmm.. that works, if you have to touch some (some< 5) boards. But If you have to touch> 5 boards, this gets unhandy...

How about:

MAKEALL --check-boards -s at91

;)

;-)

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

but when you see the last test report is more than 5 years old, you will probably rather decide to initiate a code removal process.

Why not save the SHA1 with the build-/runtime-tested information? Then we could easily build helper scripts to query that database when this board was last tested.

Ok, also good idea.

Quoted 11 lines
Quoted 7 lines

If we decide to delete older boards after n release cycles without testreports, we must not decide nor look in a database. We are sure, we have only "good and working" boards ... and we just do the necessary work for new features ... and we are sure, that we get back testreports within n release cycles ...

So let us decide first, if we want to go this way ...

Yes, we should introduce some mechanism to check when a specific board was last runtime tested. But I fear the overhead with patches that update a tested counter.

I thought with "decide": Do we want to delete "old boards"? With this, we do not need a "MAKEALL --check-boards -s at91" when we introduce new features, as all boards in mainline are in a well tested shape ...

Ok, two decisions:

- Do we want to collect board testinginformation?
- Do we want to delete old boards automatically after we do not get some test reports after a time intervall?

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#6 lore raw

Hi Heiko,

Quoted 3 lines · Heiko Schocher
Quoted 3 lines
Quoted 3 lines

Right, boards.cfg gets unhandy ... Hmm .. what with the column "Staus" ... instead of "Active" it would be more informative to have there the livetime counter, and a single digit saves some characters ;-)

I'm not sure we need to save characters from boards.cfg.

Note also that one goal of boards.cfg is to not have multiple files around that have to remain consistent.

Still :

Quoted 3 lines
Quoted 1 line

I can't understand the status field at all, just for the record ;)

Hmm.. good question ...

There has been some discussion on this already; indeed, the status field is not really aptly named or defined and should be reworked.

We could, for instance, have a "Last tested" field instead of the "Active" field.

Quoted 3 lines

Ok, two decisions:

- Do we want to collect board testinginformation?

I'd say we might want to, if only because it tells us if a board maintainer is still active or not, instead of finding out long later.

Quoted 2 lines

- Do we want to delete old boards automatically after we do not get some test reports after a time intervall?

I don't, at least, not without a long enough period during which the board remains in "unmaintained" state. E.g., for each release, the list of "unmaintained" boards is produced along with the release notes, and boards which are still "unmaintained" when nearing the next release (roughly 90 days later) get deleted right before the new release.

Quoted 2 lines

bye,
Heiko

Amicalement,

Signature · 1 line
Albert.
#7 lore raw

Hello Albert,

Am 07.11.2013 12:13, schrieb Albert ARIBAUD:

Quoted 5 lines

Hi Heiko,

On Thu, 07 Nov 2013 11:39:08 +0100, Heiko Schocher<[email protected]> wrote:

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Right, boards.cfg gets unhandy ... Hmm .. what with the column "Staus" ... instead of "Active" it would be more informative to have there the livetime counter, and a single digit saves some characters ;-)

I'm not sure we need to save characters from boards.cfg.

Note also that one goal of boards.cfg is to not have multiple files around that have to remain consistent.

Yep, exactly. Thats why I think we could collect this in boards.cfg.

Quoted 11 lines

Still :

Quoted 3 lines
Quoted 1 line

I can't understand the status field at all, just for the record ;)

Hmm.. good question ...

There has been some discussion on this already; indeed, the status field is not really aptly named or defined and should be reworked.

We could, for instance, have a "Last tested" field instead of the "Active" field.

Yep.

Quoted 6 lines
Quoted 3 lines

Ok, two decisions:

- Do we want to collect board testinginformation?

I'd say we might want to, if only because it tells us if a board maintainer is still active or not, instead of finding out long later.

Yes.

Quoted 8 lines
Quoted 2 lines

- Do we want to delete old boards automatically after we do not get some test reports after a time intervall?

I don't, at least, not without a long enough period during which the board remains in "unmaintained" state. E.g., for each release, the list of "unmaintained" boards is produced along with the release notes, and boards which are still "unmaintained" when nearing the next release (roughly 90 days later) get deleted right before the new release.

I proposed:

livetime = 4 releases cycle without testing
If livetime = 0 -> EMail to board maintainer, please test or board get deleted.

If livetime -1 -> board deleted....

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#8 lore raw

Dear Heiko Schocher,

Quoted 5 lines · In message <[email protected]> you wrote:
Quoted 3 lines

Note also that one goal of boards.cfg is to not have multiple files around that have to remain consistent.

Yep, exactly. Thats why I think we could collect this in boards.cfg.

No, we really want to have a database here, which collects more than just a timestamp. I would lile to be able to see the history as well as information which exact commit ID has been tested (and I strongly vote to allow testing at any time, not only for a end-of-release- cycle version). we should probably also collect information about the build environment (at least versions of make, gcc, bintuils), etc.

This by far exceeds what could be done in boards.cfg, and it exceeds anything that could be run over the mailing list.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
The important thing about being a leader is not being right or wrong,
but being *certain*.                    - Terry Pratchett, _Truckers_
#9 lore raw

Hello Wolfgang,

Am 07.11.2013 13:06, schrieb Wolfgang Denk:

Quoted 16 lines

Dear Heiko Schocher,

Quoted 5 lines · In message<[email protected]> you wrote:
Quoted 3 lines

Note also that one goal of boards.cfg is to not have multiple files around that have to remain consistent.

Yep, exactly. Thats why I think we could collect this in boards.cfg.

No, we really want to have a database here, which collects more than just a timestamp. I would lile to be able to see the history as well as information which exact commit ID has been tested (and I strongly vote to allow testing at any time, not only for a end-of-release- cycle version). we should probably also collect information about the build environment (at least versions of make, gcc, bintuils), etc.

Quoted 2 lines

This by far exceeds what could be done in boards.cfg, and it exceeds anything that could be run over the mailing list.

Ok, if we need all such information, it breaks boards.cfg, correct.

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#10 lore raw
In reply to Heiko Schocher

Hello Heiko,

Quoted 4 lines · Heiko Schocher

Am 07.11.2013 10:37, schrieb Andreas Bie?mann:

Quoted 3 lines · Heiko Schocher
Quoted 2 lines

Am 06.11.2013 08:50, schrieb Wolfgang Denk:

Quoted 1 line

In message<20131105203736.GM5925@bill-the-cat> you wrote:

<snip>

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines

But you are right, that approach leads in a lot of conflicting patches ... but I think, we just pooled board information in boards.cfg, so this would be the right place in my eyes ...

Maybe we get such Information "a Boards is tested with current mainline" inform of an EMail with an Text "Board xy tested with commit mm. Please update livetime" ... and we can add a script, which updates the livetime for this board, so we can prevent conflicting patches ... ?

Quoted 8 lines

I agree here with Tom. Beside the possibility of conflicting pahces I see another problem here.
We will get a lot of patches just for increasing the tested counter for a single board. These patches needs to be handled in some way. If we shift to some integrated system (gerrit comes to mind) this could be easier than today, but it will bind resources anyways. Therefore I think it is a bad idea to save a such often changing information in the source code repository.

Quoted 1 line

I see this info just changing once when releasing a new U-Boot version.

The saved information how often a board was runtime tested with the correct SHA1 of the u-boot/master could be quite useful.
In the end just the last tested commit will be interesting but it could give some information how often that specific board is used. The information must not be generated by a board maintainer ... the maintainer could then see if he needs to pull out a board or if one else run the test before.

If we would save this in the repository we do not have this information in time.
If we send the information to a list we need to parse it or use some other tool to provide the information.
Beside that we will pollute the list with status updates about boards being tested. It could be hard to find real patches in that information flood then.

<snip mail proposal>

Quoted 6 lines

So (in current case Tom) should, before releasing a new U-Boot version, first call this script "collect_livetime_info" and he get:

-> one livetime counter patch for current release
-> one list for boards which reach end of life
-> one list for boards, which should be deleted

Good idea, but the information could also be saved on a website or in another database.
It should be easily filled by the tester and also easily queried by wherever is interested in.

Quoted 2 lines

All Infos are "release info" I think, and fully fit in the commit for the new release ...

I also think that should be done on release only.

Quoted 2 lines

... maybe "deleting boards" can be done automatically, but this is not a trivial job ...

I think deleting should be done in next release then to give the board maintainer some time to check the boards. On a new release the board maintainer should be mailed that in the next release the board will be removed. We should also store this somewhere in the code (status in boards.cfg?).

Next question is what to do if the mail bounces ;)

Quoted 6 lines

So, with such a solution, I see no big additional cost for adding such a feature (except the task "deleting old boards", which is maybe not trivial)

Do not understand me wrong, if we find another solution, I am happy also ... just spinning around ...

Me too.

<snip>

Quoted 12 lines
Quoted 11 lines
Quoted 7 lines

If we decide to delete older boards after n release cycles without testreports, we must not decide nor look in a database. We are sure, we have only "good and working" boards ... and we just do the necessary work for new features ... and we are sure, that we get back testreports within n release cycles ...

So let us decide first, if we want to go this way ...

Yes, we should introduce some mechanism to check when a specific board was last runtime tested. But I fear the overhead with patches that update a tested counter.

Quoted 8 lines

I thought with "decide": Do we want to delete "old boards"? With this, we do not need a "MAKEALL --check-boards -s at91" when we introduce new features, as all boards in mainline are in a well tested shape ...

Ok, two decisions:

- Do we want to collect board testinginformation?

I think we should do that i none way or another.

Quoted 2 lines

- Do we want to delete old boards automatically after we do not get some test reports after a time intervall?

And we should delete 'unmaintained' boards, when is to be discussed. I'm currently fiddling with at91 gpio and ask myself if I should adopt all the boards or just let them fail ...

Best regards

Andreas Bie?mann

#11 lore raw

Hello Andreas,

Am 07.11.2013 12:24, schrieb Andreas Bie?mann:

Quoted 38 lines

Hello Heiko,

Quoted 4 lines · Heiko Schocher

Am 07.11.2013 10:37, schrieb Andreas Bie?mann:

Quoted 3 lines · Heiko Schocher
Quoted 2 lines

Am 06.11.2013 08:50, schrieb Wolfgang Denk:

Quoted 1 line

In message<20131105203736.GM5925@bill-the-cat> you wrote:

<snip>

Quoted 18 lines
Quoted 9 lines
Quoted 8 lines

But you are right, that approach leads in a lot of conflicting patches ... but I think, we just pooled board information in boards.cfg, so this would be the right place in my eyes ...

Maybe we get such Information "a Boards is tested with current mainline" inform of an EMail with an Text "Board xy tested with commit mm. Please update livetime" ... and we can add a script, which updates the livetime for this board, so we can prevent conflicting patches ... ?

Quoted 8 lines

I agree here with Tom. Beside the possibility of conflicting pahces I see another problem here.
We will get a lot of patches just for increasing the tested counter for a single board. These patches needs to be handled in some way. If we shift to some integrated system (gerrit comes to mind) this could be easier than today, but it will bind resources anyways. Therefore I think it is a bad idea to save a such often changing information in the source code repository.

Quoted 1 line

I see this info just changing once when releasing a new U-Boot version.

The saved information how often a board was runtime tested with the correct SHA1 of the u-boot/master could be quite useful.
In the end just the last tested commit will be interesting but it could give some information how often that specific board is used. The information must not be generated by a board maintainer ... the maintainer could then see if he needs to pull out a board or if one else run the test before.

Quoted 7 lines

If we would save this in the repository we do not have this information in time.
If we send the information to a list we need to parse it or use some other tool to provide the information.
Beside that we will pollute the list with status updates about boards being tested. It could be hard to find real patches in that information flood then.

Hmm... I hope we get a lot of such EMails ... and think, this is not a big problem ... Or, maybe, if we get a lot of such EMails, maybe we open a u-boot-testing list?

Quoted 5 lines

<snip mail proposal>

Quoted 3 lines

So (in current case Tom) should, before releasing a new U-Boot version, first call this script "collect_livetime_info" and he get:

Quoted 8 lines
Quoted 3 lines
->  one livetime counter patch for current release
->  one list for boards which reach end of life
->  one list for boards, which should be deleted

Good idea, but the information could also be saved on a website or in another database.
It should be easily filled by the tester and also easily queried by wherever is interested in.

Ok, if we have this info, we can show it wherever we want ...

Quoted 4 lines
Quoted 2 lines

All Infos are "release info" I think, and fully fit in the commit for the new release ...

I also think that should be done on release only.

Yep! But collecting this infos can be done all the time.

Quoted 8 lines
Quoted 2 lines

... maybe "deleting boards" can be done automatically, but this is not a trivial job ...

I think deleting should be done in next release then to give the board maintainer some time to check the boards. On a new release the board maintainer should be mailed that in the next release the board will be removed. We should also store this somewhere in the code (status in boards.cfg?).

See my proposal for the livetime counter:

livetime init value n (n=4)
livetimer decrement on every new release
livetimer set to n, if in release cycle comes a test report livetimer == 0 -> EMail to board maintainer, board reached end of live in
mainline, please send test report.
livetimer == -1 -> board get deleted

So all info is in boards.cfg availiable ...

Quoted 1 line

Next question is what to do if the mail bounces ;)

Board gets deleted, as board maintainer didn;t send an update patch for boards.cfg ...

Quoted 28 lines
Quoted 6 lines

So, with such a solution, I see no big additional cost for adding such a feature (except the task "deleting old boards", which is maybe not trivial)

Do not understand me wrong, if we find another solution, I am happy also ... just spinning around ...

Me too.

<snip>

Quoted 17 lines
Quoted 11 lines
Quoted 7 lines

If we decide to delete older boards after n release cycles without testreports, we must not decide nor look in a database. We are sure, we have only "good and working" boards ... and we just do the necessary work for new features ... and we are sure, that we get back testreports within n release cycles ...

So let us decide first, if we want to go this way ...

Yes, we should introduce some mechanism to check when a specific board was last runtime tested. But I fear the overhead with patches that update a tested counter.

I thought with "decide": Do we want to delete "old boards"? With this, we do not need a "MAKEALL --check-boards -s at91" when we introduce new features, as all boards in mainline are in a well tested shape ...

Quoted 5 lines
Quoted 3 lines

Ok, two decisions:

- Do we want to collect board testinginformation?

I think we should do that i none way or another.

Yep.

Quoted 6 lines
Quoted 2 lines

- Do we want to delete old boards automatically after we do not get some test reports after a time intervall?

And we should delete 'unmaintained' boards, when is to be discussed. I'm currently fiddling with at91 gpio and ask myself if I should adopt all the boards or just let them fail ...

You do not have this problem when we descide to delete old boards!

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#12 lore raw

Dear Heiko,

Quoted 4 lines · In message <[email protected]> you wrote:

Hmm... I hope we get a lot of such EMails ... and think, this is not a big problem ... Or, maybe, if we get a lot of such EMails, maybe we open a u-boot-testing list?

NAK. Mailing lists are good for some kind of information - especially for such information that is read by humans.

All you want to do here is feed a database with data. This is not what mailing lists were made for, so we should really use a more appropriate interface.

Quoted 8 lines

See my proposal for the livetime counter:

livetime init value n (n=4)
livetimer decrement on every new release
livetimer set to n, if in release cycle comes a test report livetimer == 0 -> EMail to board maintainer, board reached end of live in
mainline, please send test report.
livetimer == -1 -> board get deleted

This is too simple in one respect (as you are not including information that may be vital, like which exact commit ID has been tested, or which tool chain has been used to build it).

Also we should probably not only allow for positive test reports, but also allow to report failures (to mark board support as broken).

All this cannot be done in boards.cfg .

And there is no need to as long as we provide tools to query for any information we might be interested in.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Every living thing wants to survive.
	-- Spock, "The Ultimate Computer", stardate 4731.3
#13 lore raw

Hello Wolfgang,

Am 07.11.2013 13:12, schrieb Wolfgang Denk:

Quoted 8 lines

Dear Heiko,

Quoted 4 lines · In message<[email protected]> you wrote:

Hmm... I hope we get a lot of such EMails ... and think, this is not a big problem ... Or, maybe, if we get a lot of such EMails, maybe we open a u-boot-testing list?

Quoted 6 lines

NAK. Mailing lists are good for some kind of information - especially for such information that is read by humans.

All you want to do here is feed a database with data. This is not what mailing lists were made for, so we should really use a more appropriate interface.

Ok. But this status report can be in readable text format ;-)

Quoted 9 lines
Quoted 8 lines

See my proposal for the livetime counter:

livetime init value n (n=4)
livetimer decrement on every new release
livetimer set to n, if in release cycle comes a test report livetimer == 0 -> EMail to board maintainer, board reached end of live in
mainline, please send test report.
livetimer == -1 -> board get deleted

Quoted 3 lines

This is too simple in one respect (as you are not including information that may be vital, like which exact commit ID has been tested, or which tool chain has been used to build it).

Ok.

Quoted 2 lines

Also we should probably not only allow for positive test reports, but also allow to report failures (to mark board support as broken).

Yep, this will set livetime = 0 -> EMail to board maintainer ...

Quoted 4 lines

All this cannot be done in boards.cfg .

And there is no need to as long as we provide tools to query for any information we might be interested in.

Ok, then we must define this way ...

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#14 lore raw

Dear Heiko Schocher,

Quoted 6 lines · In message <[email protected]> you wrote:
Quoted 4 lines

All you want to do here is feed a database with data. This is not what mailing lists were made for, so we should really use a more appropriate interface.

Ok. But this status report can be in readable text format ;-)

Yes, it can. So do you want to see 250 "test passed" messages for the BeagleBone or the Rasperry Pi for every few commits that get merged? I don't. In a database, we can automatically filter redundant information.

You, you can use a mailing list for submitting such information, but I doubt that it would be efficient. And I definitely do not want to see this on the current U-Boot ML.

Best regards,

Wolfgang Denk

Signature · 7 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
"You ain't experienced..." "Well, nor are you." "That's true. But the
point is ... the point is ... the point is we've been not experienced
for a lot longer than you. We've got  a  lot  of  experience  of  not
having any experience."           - Terry Pratchett, _Witches Abroad_
#15 lore raw

Hello Wolfgang,

Am 07.11.2013 20:19, schrieb Wolfgang Denk:

Quoted 8 lines

Dear Heiko Schocher,

Quoted 5 lines · In message<[email protected]> you wrote:
Quoted 4 lines

All you want to do here is feed a database with data. This is not what mailing lists were made for, so we should really use a more appropriate interface.

Quoted 6 lines
Quoted 1 line

Ok. But this status report can be in readable text format ;-)

Yes, it can. So do you want to see 250 "test passed" messages for the BeagleBone or the Rasperry Pi for every few commits that get merged? I don't. In a database, we can automatically filter redundant information.

Ok, thats a good point!

Quoted 3 lines

You, you can use a mailing list for submitting such information, but I doubt that it would be efficient. And I definitely do not want to see this on the current U-Boot ML.

Ok, let us discuss the way we collect such information, if we decided that we want testreports ...

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#16 lore raw

Dear Heiko,

Quoted 7 lines · In message <[email protected]> you wrote:
Quoted 4 lines

You, you can use a mailing list for submitting such information, but I doubt that it would be efficient. And I definitely do not want to see this on the current U-Boot ML.

Ok, let us discuss the way we collect such information, if we decided that we want testreports ...

I think there is a general agreement that we do want such reports.

We just need to get a better understanding, what exactly we want, and then how we can get this.

Best regards,

Wolfgang Denk

Signature · 4 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Time is an illusion perpetrated by the manufacturers of space.
#17 lore raw
In reply to Andreas Bießmann

Dear "Andreas Bie?mann",

Quoted 6 lines · In message <[email protected]> you wrote:

The saved information how often a board was runtime tested with the correct SHA1 of the u-boot/master could be quite useful.
In the end just the last tested commit will be interesting but it could give some information how often that specific board is used. The information must not be generated by a board maintainer ... the

s/must not/need not/ (faux ami in action, I guess)

Quoted 2 lines

maintainer could then see if he needs to pull out a board or if one else run the test before.

I fully agree - everybody should be able to provide such test information. Actually it would be a big help to board maintainers as well if these would get test reports from actual users of the hardware.

Quoted 7 lines

If we would save this in the repository we do not have this information in time.
If we send the information to a list we need to parse it or use some other tool to provide the information.
Beside that we will pollute the list with status updates about boards being tested. It could be hard to find real patches in that information flood then.

Agreed too. I doubt if a mailing list makes sense to collect such data. It would probably be more efficient to provide a web based service for this. It just has to be easy to submit reports, and to query the status for boards.

Quoted 8 lines
Quoted 3 lines

-> one livetime counter patch for current release
-> one list for boards which reach end of life
-> one list for boards, which should be deleted

Good idea, but the information could also be saved on a website or in another database.
It should be easily filled by the tester and also easily queried by wherever is interested in.

Agreed. I definitely do not want to see such trafic on the regular U-Boot mailing list.

Quoted 4 lines
Quoted 2 lines

All Infos are "release info" I think, and fully fit in the commit for the new release ...

I also think that should be done on release only.

Why? To me it makes a lot of sense to also collect information on intermediate snapshots.

Quoted 7 lines

I think deleting should be done in next release then to give the board maintainer some time to check the boards. On a new release the board maintainer should be mailed that in the next release the board will be removed. We should also store this somewhere in the code (status in boards.cfg?).

Next question is what to do if the mail bounces ;)

Mail bounces (and new address of maintainer cannot be found, and no other user volunteers to take over maintenance) => board is unmaintained => board gets removed.

Quoted 6 lines
Quoted 2 lines

- Do we want to delete old boards automatically after we do not get some test reports after a time intervall?

And we should delete 'unmaintained' boards, when is to be discussed. I'm currently fiddling with at91 gpio and ask myself if I should adopt all the boards or just let them fail ...

I hesitate to automatically remove existing boards. Why would we want to do that? To reduce efforts, right? So I vote to keep boards as long as they are either maintained, or they at least "do not hurt". If a board just builds fine and does not cause any additional efforts we should keep it, no matter if there is an active maintainer or test reports or not. Only when a board becomes a pain to somebody - say, because it develops build errors, or wit would require efforts to adapt it to some new feature, _then_ we would check if this is one of the "precious" boards we want to keep or if it is just old cruft nobody cares about anyway. And only then I would remove it.

Best regards,

Wolfgang Denk

Signature · 5 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Simplicity boils down to two steps: Identify the essential. Eliminate
the rest.                                               - Leo Babauta
#18 lore raw

Dear Wolfgang Denk,

Quoted 9 lines · Wolfgang Denk
Quoted 6 lines · In message <[email protected]> you wrote:

The saved information how often a board was runtime tested with the correct SHA1 of the u-boot/master could be quite useful.
In the end just the last tested commit will be interesting but it could give some information how often that specific board is used. The information must not be generated by a board maintainer ... the

s/must not/need not/ (faux ami in action, I guess)

You're right

<snip>

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

All Infos are "release info" I think, and fully fit in the commit for the new release ...

I also think that should be done on release only.

Why? To me it makes a lot of sense to also collect information on intermediate snapshots.

Well, I think we should query that database on release and transform the testing information to some information maybe stored in release code and to trigger board maintainers to do tests.
Maybe the proposed lifetime counter is the correct tooling here.

We should introduce some service to gather testing information which should have quite high throughput.
But we should also install some measures to see the 'liveliness' of some board's tests in the released source code.

Quoted 11 lines
Quoted 7 lines

I think deleting should be done in next release then to give the board maintainer some time to check the boards. On a new release the board maintainer should be mailed that in the next release the board will be removed. We should also store this somewhere in the code (status in boards.cfg?).

Next question is what to do if the mail bounces ;)

Mail bounces (and new address of maintainer cannot be found, and no other user volunteers to take over maintenance) => board is unmaintained => board gets removed.

Sounds good, but doesn't fit to your next statement ...

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

- Do we want to delete old boards automatically after we do not get some test reports after a time intervall?

And we should delete 'unmaintained' boards, when is to be discussed. I'm currently fiddling with at91 gpio and ask myself if I should adopt all the boards or just let them fail ...

Quoted 10 lines

I hesitate to automatically remove existing boards. Why would we want to do that? To reduce efforts, right? So I vote to keep boards as long as they are either maintained, or they at least "do not hurt". If a board just builds fine and does not cause any additional efforts we should keep it, no matter if there is an active maintainer or test reports or not. Only when a board becomes a pain to somebody - say, because it develops build errors, or wit would require efforts to adapt it to some new feature, _then_ we would check if this is one of the "precious" boards we want to keep or if it is just old cruft nobody cares about anyway. And only then I would remove it.

Sounds good.

Best regards

Andreas Bie?mann

#19 lore raw
In reply to Wolfgang Denk

Hello Wolfgang,

Am 07.11.2013 13:01, schrieb Wolfgang Denk:

Quoted 4 lines

Dear "Andreas Bie?mann",

Quoted 1 line · In message<[email protected]> you wrote:

[...]

Quoted 7 lines
Quoted 2 lines

maintainer could then see if he needs to pull out a board or if one else run the test before.

I fully agree - everybody should be able to provide such test information. Actually it would be a big help to board maintainers as well if these would get test reports from actual users of the hardware.

Yep.

Quoted 8 lines
Quoted 7 lines

If we would save this in the repository we do not have this information in time.
If we send the information to a list we need to parse it or use some other tool to provide the information.
Beside that we will pollute the list with status updates about boards being tested. It could be hard to find real patches in that information flood then.

Quoted 4 lines

Agreed too. I doubt if a mailing list makes sense to collect such data. It would probably be more efficient to provide a web based service for this. It just has to be easy to submit reports, and to query the status for boards.

Ok, how would look like such a web based service?

Quoted 11 lines
Quoted 8 lines
Quoted 3 lines
->  one livetime counter patch for current release
->  one list for boards which reach end of life
->  one list for boards, which should be deleted

Good idea, but the information could also be saved on a website or in another database.
It should be easily filled by the tester and also easily queried by wherever is interested in.

Agreed. I definitely do not want to see such trafic on the regular U-Boot mailing list.

Oh, ok ... then we must look for another way.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

All Infos are "release info" I think, and fully fit in the commit for the new release ...

I also think that should be done on release only.

Why? To me it makes a lot of sense to also collect information on intermediate snapshots.

Hmm.. I thought we get a "test report" (however this looks like in the end) based on a commit id in u-boot/master ... isn;t this enough?

Quoted 11 lines
Quoted 7 lines

I think deleting should be done in next release then to give the board maintainer some time to check the boards. On a new release the board maintainer should be mailed that in the next release the board will be removed. We should also store this somewhere in the code (status in boards.cfg?).

Next question is what to do if the mail bounces ;)

Mail bounces (and new address of maintainer cannot be found, and no other user volunteers to take over maintenance) => board is unmaintained => board gets removed.

Full Ack.

Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

- Do we want to delete old boards automatically after we do not get some test reports after a time intervall?

And we should delete 'unmaintained' boards, when is to be discussed. I'm currently fiddling with at91 gpio and ask myself if I should adopt all the boards or just let them fail ...

I hesitate to automatically remove existing boards. Why would we want to do that? To reduce efforts, right? So I vote to keep boards as long as they are either maintained, or they at least "do not hurt".

Why the "not hurt case"? Is it really good to have a lot of boards, which compile clean, but we do not know, if the code really works?

I prefer to have in current mainline only boards, which really work or at least maintained... if a board maintainer did the work to bring it into mainline, it should be interested in "stay in" mainline. If this interest is lost and no other volunteers ... board is useless in mainline.

Code for old boards is not lost.

Quoted 7 lines

If a board just builds fine and does not cause any additional efforts we should keep it, no matter if there is an active maintainer or test reports or not. Only when a board becomes a pain to somebody - say, because it develops build errors, or wit would require efforts to adapt it to some new feature, _then_ we would check if this is one of the "precious" boards we want to keep or if it is just old cruft nobody cares about anyway. And only then I would remove it.

Ok, that is a way to go, if we have for all boards the information, in which maintainstate it is ... but think of for example the new i2c framework, how much boards use I2C? I have to check now all this boards, decide to delete it or convert it ... very time consuming frustrating work ... and maybe for a lot of "do not hurt" boards only waste of time ... :-(

if we have a clean mainline state, where I know all boards are working the decision is clear, convert all ... and board maintainers will test it ... and we have always a clean compile state for mainline

And if we have this test/delete cycle, I can be sure, that after a defined time all boards in mainline are working!

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#20 lore raw

Dear Heiko,

Quoted 7 lines · In message <[email protected]> you wrote:
Quoted 5 lines

Agreed too. I doubt if a mailing list makes sense to collect such data. It would probably be more efficient to provide a web based service for this. It just has to be easy to submit reports, and to query the status for boards.

Ok, how would look like such a web based service?

I have no idea. Not yet. Let's first define what we want to have, then think about how to implement it.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

I also think that should be done on release only.

Why? To me it makes a lot of sense to also collect information on intermediate snapshots.

Hmm.. I thought we get a "test report" (however this looks like in the end) based on a commit id in u-boot/master ... isn;t this enough?

Yes, this is perfectly fine. I just want to allow this at _any_ time, not only once per release (near the end of the release cycle). especially for releases where bigger changes get merged it may be precious information to know when the code stopped working.

Quoted 6 lines
Quoted 3 lines

I hesitate to automatically remove existing boards. Why would we want to do that? To reduce efforts, right? So I vote to keep boards as long as they are either maintained, or they at least "do not hurt".

Why the "not hurt case"? Is it really good to have a lot of boards, which compile clean, but we do not know, if the code really works?

Well, one reason is efficiency. If the code builds fine, and does not cause efforts druing any of the ongoing work, it is more efficient to just keep it than to actively remove it, which would require active work. I. e. I want to reduce the work load on the maintainer(s) such that they only have to become active if such action saves even more effort. Actually it may even seem more efficient in some cases to perform minor fixing even of unmaintained boards than to remove them.

Quoted 5 lines

I prefer to have in current mainline only boards, which really work or at least maintained... if a board maintainer did the work to bring it into mainline, it should be interested in "stay in" mainline. If this interest is lost and no other volunteers ... board is useless in mainline.

But should we not also try to minimize efforts, especially on the custodians? If a board does not cause any trouble, we should not have to invent additional efforts for it.

Quoted 6 lines

Ok, that is a way to go, if we have for all boards the information, in which maintainstate it is ... but think of for example the new i2c framework, how much boards use I2C? I have to check now all this boards, decide to delete it or convert it ... very time consuming frustrating work ... and maybe for a lot of "do not hurt" boards only waste of time ... :-(

I don't really understand this argument. The I2C code is either hardware independent (say, the command line interface code), or it is platform specific (say, the driver code for the I2C controller on a specific SoC), or it is board dendent (say, some specific twiddeling with I2C devices to perform some magic operations on a board).

In the first two cases the work wil have to be done in any case (except for the realy rare case that there are only old, unmaintained boards left using this SoC). An for the board specific code you can check if you need to adapt it by checking the board's test status. If the board is unmaintained, then we enter the "it hurts" branch, and drop the board.

Quoted 3 lines

if we have a clean mainline state, where I know all boards are working the decision is clear, convert all ... and board maintainers will test it ... and we have always a clean compile state for mainline

Again: you don't need any knowledge about boards that are not affected by your I2C changes.

Quoted 2 lines

And if we have this test/delete cycle, I can be sure, that after a defined time all boards in mainline are working!

Yes, but the cost is additional efforts, and being more aggressive than necessary. I dislike both.

Best regards,

Wolfgang Denk

Signature · 4 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Life would be so much easier if everyone read the manual.
#21 lore raw

Hello Wolfgang,

Am 07.11.2013 20:15, schrieb Wolfgang Denk:

Quoted 9 lines

Dear Heiko,

Quoted 6 lines · In message<[email protected]> you wrote:
Quoted 5 lines

Agreed too. I doubt if a mailing list makes sense to collect such data. It would probably be more efficient to provide a web based service for this. It just has to be easy to submit reports, and to query the status for boards.

Quoted 4 lines
Quoted 1 line

Ok, how would look like such a web based service?

I have no idea. Not yet. Let's first define what we want to have, then think about how to implement it.

Ok.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 1 line

I also think that should be done on release only.

Why? To me it makes a lot of sense to also collect information on intermediate snapshots.

Quoted 7 lines
Quoted 2 lines

Hmm.. I thought we get a "test report" (however this looks like in the end) based on a commit id in u-boot/master ... isn;t this enough?

Yes, this is perfectly fine. I just want to allow this at _any_ time, not only once per release (near the end of the release cycle). especially for releases where bigger changes get merged it may be precious information to know when the code stopped working.

With "once", I only meant, we once update the "livetimer" and collect all the time test reports!

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I hesitate to automatically remove existing boards. Why would we want to do that? To reduce efforts, right? So I vote to keep boards as long as they are either maintained, or they at least "do not hurt".

Why the "not hurt case"? Is it really good to have a lot of boards, which compile clean, but we do not know, if the code really works?

Quoted 7 lines

Well, one reason is efficiency. If the code builds fine, and does not cause efforts druing any of the ongoing work, it is more efficient to just keep it than to actively remove it, which would require active work. I. e. I want to reduce the work load on the maintainer(s) such that they only have to become active if such action saves even more effort. Actually it may even seem more efficient in some cases to perform minor fixing even of unmaintained boards than to remove them.

But in the long term I think, we have more work with old, unmaintained boards, then removing it. And if we remove boards actively, we maybe force more the board maintainers (for examples corporates who have interest that the board stays in mainline) to really test them ;-)

Quoted 9 lines
Quoted 5 lines

I prefer to have in current mainline only boards, which really work or at least maintained... if a board maintainer did the work to bring it into mainline, it should be interested in "stay in" mainline. If this interest is lost and no other volunteers ... board is useless in mainline.

But should we not also try to minimize efforts, especially on the custodians? If a board does not cause any trouble, we should not have to invent additional efforts for it.

Hmm... I am not really sure, if it is more work for removing a board or fixing compiling issues ...

Quoted 13 lines
Quoted 6 lines

Ok, that is a way to go, if we have for all boards the information, in which maintainstate it is ... but think of for example the new i2c framework, how much boards use I2C? I have to check now all this boards, decide to delete it or convert it ... very time consuming frustrating work ... and maybe for a lot of "do not hurt" boards only waste of time ... :-(

I don't really understand this argument. The I2C code is either hardware independent (say, the command line interface code), or it is platform specific (say, the driver code for the I2C controller on a specific SoC), or it is board dendent (say, some specific twiddeling with I2C devices to perform some magic operations on a board).

Quoted 7 lines

In the first two cases the work wil have to be done in any case (except for the realy rare case that there are only old, unmaintained boards left using this SoC). An for the board specific code you can

check if you need to adapt it by checking the board's test status. If the board is unmaintained, then we enter the "it hurts" branch, and drop the board.

Hmm... :

pollux:u-boot hs [master] $ grep -lr HARD_I2C arch/powerpc/ [...]
arch/powerpc/cpu/mpc8xx/i2c.c
[...]
arch/powerpc/cpu/mpc8260/i2c.c arch/powerpc/cpu/mpc5xxx/i2c.c arch/powerpc/cpu/mpc824x/drivers/i2c/i2c.c arch/powerpc/cpu/mpc512x/i2c.c
pollux:u-boot hs [master] $

Is it worth to move the drivers to drivers/i2c/ and convert them to the new framework? Or is it better to delete them and the boards, which use them ... Ok, if we had such a "livetimer" without deleting old boards, you are right, I could do now:

- search the boards, which use the drivers
- search the state of this board
- decide, dropping or convert ...
On which criteria whould this decision be done? I have no idea if this boards exist anymore ...

If we have only good boards, I have only one way -> convert. No time wasted for the above steps ...

So, yes, I think we do not need to delete the boards, just mark it broken ...

My hope with "delete old boards from mainline" is, that we become more testreports from board maintainers, as they (should) have interest that the board support stays in mainline. If they know, that the board is only marked as broken, old or whatever ... they maybe do not reserve the time for testing a board once a year ...

Quoted 4 lines
Quoted 3 lines

if we have a clean mainline state, where I know all boards are working the decision is clear, convert all ... and board maintainers will test it ... and we have always a clean compile state for mainline

Quoted 8 lines

Again: you don't need any knowledge about boards that are not affected by your I2C changes.

Quoted 2 lines

And if we have this test/delete cycle, I can be sure, that after a defined time all boards in mainline are working!

Yes, but the cost is additional efforts, and being more aggressive than necessary. I dislike both.

I dislike additional efforts too (But the question is, needs removing old boards more effort than fixing them all the time maybe uselessly?). Is it really aggressive to want once a year a testreport from board maintainers? They get a lot of effort from the mainline for free...

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#22 lore raw

Dear Heiko,

Quoted 8 lines · In message <[email protected]> you wrote:
Quoted 5 lines

Yes, this is perfectly fine. I just want to allow this at _any_ time, not only once per release (near the end of the release cycle). especially for releases where bigger changes get merged it may be precious information to know when the code stopped working.

With "once", I only meant, we once update the "livetimer" and collect all the time test reports!

But with your suggestion, the only stoprage for such information is
the life timer entry in the boards.cfg file. Or would you suggest to
store additional information elsewhere? Then we don't need the (then redundant) counter in boards.cfg ...

Quoted 10 lines
Quoted 7 lines

Well, one reason is efficiency. If the code builds fine, and does not cause efforts druing any of the ongoing work, it is more efficient to just keep it than to actively remove it, which would require active work. I. e. I want to reduce the work load on the maintainer(s) such that they only have to become active if such action saves even more effort. Actually it may even seem more efficient in some cases to perform minor fixing even of unmaintained boards than to remove them.

But in the long term I think, we have more work with old, unmaintained boards, then removing it. And if we remove boards actively, we maybe

YOu misunderstand. My intention is to keep them as long as they do NOT cause any efforts. If they do, they are candidates for checking for removal.

Quoted 2 lines

force more the board maintainers (for examples corporates who have interest that the board stays in mainline) to really test them ;-)

Yes, but you organize more work for all of us.

Quoted 6 lines
Quoted 3 lines

But should we not also try to minimize efforts, especially on the custodians? If a board does not cause any trouble, we should not have to invent additional efforts for it.

Hmm... I am not really sure, if it is more work for removing a board or fixing compiling issues ...

If we have to fix compilation issues, that means that the board _IS_ a candidate for removal. I suggested to keep them as long as they do "not cause any trouble" !

Quoted 16 lines
Quoted 6 lines

I don't really understand this argument. The I2C code is either hardware independent (say, the command line interface code), or it is platform specific (say, the driver code for the I2C controller on a specific SoC), or it is board dendent (say, some specific twiddeling with I2C devices to perform some magic operations on a board).

Quoted 7 lines

In the first two cases the work wil have to be done in any case (except for the realy rare case that there are only old, unmaintained boards left using this SoC). An for the board specific code you can

check if you need to adapt it by checking the board's test status. If the board is unmaintained, then we enter the "it hurts" branch, and drop the board.

Hmm... :

Quoted 9 lines

pollux:u-boot hs [master] $ grep -lr HARD_I2C arch/powerpc/ [...]
arch/powerpc/cpu/mpc8xx/i2c.c
[...]
arch/powerpc/cpu/mpc8260/i2c.c arch/powerpc/cpu/mpc5xxx/i2c.c arch/powerpc/cpu/mpc824x/drivers/i2c/i2c.c arch/powerpc/cpu/mpc512x/i2c.c
pollux:u-boot hs [master] $

THi sis all architecure support. None of these files are board specific. None of them would be removed if you crap old boards.

Quoted 4 lines

Is it worth to move the drivers to drivers/i2c/ and convert them to the new framework? Or is it better to delete them and the boards, which use them ... Ok, if we had such a "livetimer" without deleting old boards, you are right, I could do now:

How do you know which boards use these drivers?

Quoted 5 lines

- search the boards, which use the drivers
- search the state of this board
- decide, dropping or convert ...
On which criteria whould this decision be done? I have no idea if this boards exist anymore ...

There is no conflict with what I wrote - if you can find that only "old" boards are using these drivers, then both the boards and the drivers are candidates for removalk. But you need to find this correlation first, and verify that there are no "good" boards left that use trhem do.

All of this is totally different of the mechanism how to determine what are "good" or "old" boards.

Quoted 2 lines

If we have only good boards, I have only one way -> convert. No time wasted for the above steps ...

It does not really matter when you have to check if there are "good" users of this code left. You have to do this in both cases - either when removing the boards automatically, or when touching that code.

With your solution, you will probably have to do it twice, as you cannot be really sure that the guy who removed the board support verified that he was the last user of some global architecture support code.

Quoted 2 lines

I dislike additional efforts too (But the question is, needs removing old boards more effort than fixing them all the time maybe uselessly?).

Can you please re-read what I wrote? I say we keep them as long as they do NOT need ANY fixing. If they do, this is a toalluy new game.

Best regards,

Wolfgang Denk

Signature · 4 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
The following statement is not true.  The previous statement is true.
#23 lore raw

Hello Wolfgang,

Am 08.11.2013 07:20, schrieb Wolfgang Denk:

Quoted 9 lines

Dear Heiko,

Quoted 6 lines · In message<[email protected]> you wrote:
Quoted 5 lines

Yes, this is perfectly fine. I just want to allow this at _any_ time, not only once per release (near the end of the release cycle). especially for releases where bigger changes get merged it may be precious information to know when the code stopped working.

Quoted 7 lines
Quoted 2 lines

With "once", I only meant, we once update the "livetimer" and collect all the time test reports!

But with your suggestion, the only stoprage for such information is
the life timer entry in the boards.cfg file. Or would you suggest to
store additional information elsewhere? Then we don't need the (then redundant) counter in boards.cfg ...

Yes, if we store more information then the "livetime", we need another approach. But with storing this somewhere else, we must take care that the number and names of the boards in the external database are in sync with the boards in mainline.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

Well, one reason is efficiency. If the code builds fine, and does not cause efforts druing any of the ongoing work, it is more efficient to just keep it than to actively remove it, which would require active work. I. e. I want to reduce the work load on the maintainer(s) such that they only have to become active if such action saves even more effort. Actually it may even seem more efficient in some cases to perform minor fixing even of unmaintained boards than to remove them.

Quoted 6 lines
Quoted 2 lines

But in the long term I think, we have more work with old, unmaintained boards, then removing it. And if we remove boards actively, we maybe

YOu misunderstand. My intention is to keep them as long as they do NOT cause any efforts. If they do, they are candidates for checking for removal.

Ah, ok!

Quoted 4 lines
Quoted 2 lines

force more the board maintainers (for examples corporates who have interest that the board stays in mainline) to really test them ;-)

Yes, but you organize more work for all of us.

That would be bad.

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

But should we not also try to minimize efforts, especially on the custodians? If a board does not cause any trouble, we should not have to invent additional efforts for it.

Hmm... I am not really sure, if it is more work for removing a board or fixing compiling issues ...

If we have to fix compilation issues, that means that the board _IS_ a candidate for removal. I suggested to keep them as long as they do "not cause any trouble" !

Ok, got it, that sounds good. So, if we introduce new features we have to check for boards which are affected:

- board is in maintained state -> convert to new feature
- board is in unmaintained state -> drop it ...

Quoted 26 lines
Quoted 16 lines
Quoted 6 lines

I don't really understand this argument. The I2C code is either hardware independent (say, the command line interface code), or it is platform specific (say, the driver code for the I2C controller on a specific SoC), or it is board dendent (say, some specific twiddeling with I2C devices to perform some magic operations on a board).

Quoted 7 lines

In the first two cases the work wil have to be done in any case (except for the realy rare case that there are only old, unmaintained boards left using this SoC). An for the board specific code you can

check if you need to adapt it by checking the board's test status. If the board is unmaintained, then we enter the "it hurts" branch, and drop the board.

Hmm... :

Quoted 9 lines

pollux:u-boot hs [master] $ grep -lr HARD_I2C arch/powerpc/ [...]
arch/powerpc/cpu/mpc8xx/i2c.c
[...]
arch/powerpc/cpu/mpc8260/i2c.c arch/powerpc/cpu/mpc5xxx/i2c.c arch/powerpc/cpu/mpc824x/drivers/i2c/i2c.c arch/powerpc/cpu/mpc512x/i2c.c
pollux:u-boot hs [master] $

Quoted 2 lines

THi sis all architecure support. None of these files are board specific. None of them would be removed if you crap old boards.

Hmm.. a U-Boot rule is, remove unnused code, or? If we remove all boards, who use this drivers, we remove the drivers also ...

Quoted 6 lines
Quoted 4 lines

Is it worth to move the drivers to drivers/i2c/ and convert them to the new framework? Or is it better to delete them and the boards, which use them ... Ok, if we had such a "livetimer" without deleting old boards, you are right, I could do now:

How do you know which boards use these drivers?

I must find out ...

Quoted 11 lines
Quoted 5 lines

- search the boards, which use the drivers
- search the state of this board
- decide, dropping or convert ...
On which criteria whould this decision be done? I have no idea if this boards exist anymore ...

There is no conflict with what I wrote - if you can find that only "old" boards are using these drivers, then both the boards and the drivers are candidates for removalk. But you need to find this correlation first, and verify that there are no "good" boards left that use trhem do.

Yes.

But there is a difference! With having old boards in mainline, I must do the above step "search the state, and decide what to do". With having only "good" boards in mainline, I do not need this steps.

But I aggree with you, if we have the rule:
"unmaintained board and compile error" -> remove it

This decision is easy and I am fine with it.

Quoted 6 lines

All of this is totally different of the mechanism how to determine what are "good" or "old" boards.

Quoted 2 lines

If we have only good boards, I have only one way -> convert. No time wasted for the above steps ...

Quoted 8 lines

It does not really matter when you have to check if there are "good" users of this code left. You have to do this in both cases - either when removing the boards automatically, or when touching that code.

With your solution, you will probably have to do it twice, as you cannot be really sure that the guy who removed the board support verified that he was the last user of some global architecture support code.

Yes for the driver code. As I said in this discusion, removing boards is no trivial job.

Quoted 5 lines
Quoted 2 lines

I dislike additional efforts too (But the question is, needs removing old boards more effort than fixing them all the time maybe uselessly?).

Can you please re-read what I wrote? I say we keep them as long as they do NOT need ANY fixing. If they do, this is a toalluy new game.

I got it now ;-)

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#24 lore raw
In reply to Andreas Bießmann
Quoted 4 lines · Andreas Bie?mann

Hello all together,

Quoted 1 line · Heiko Schocher

Am 06.11.2013 08:50, schrieb Wolfgang Denk:

[snip]

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

So when you're once again doing some change that requires touching files for some othe rboards, you could simply check that database. If you see that 3 out of the last 5 releases have reported succesful run-time tests you will probably decide to accept the needed efforts,

Quoted 8 lines
Quoted 2 lines

Hmm.. that works, if you have to touch some (some < 5) boards. But If you have to touch > 5 boards, this gets unhandy...

How about:

MAKEALL --check-boards -s at91

;)

I feel this is the hard part of the problem, and what we're glossing over. What has to be tested by the board maintainer? What are we going to leave to their discretion? Will am335x_evm not count if I don't dig up the NOR cape for it?

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20131107/79a77722/attachment.pgp>
#25 lore raw

Dear Tom Rini,

Quoted 16 lines · Tom Rini
Quoted 4 lines · Andreas Bie?mann

Hello all together,

Quoted 1 line · Heiko Schocher

Am 06.11.2013 08:50, schrieb Wolfgang Denk:

[snip]

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

So when you're once again doing some change that requires touching files for some othe rboards, you could simply check that database. If you see that 3 out of the last 5 releases have reported succesful run-time tests you will probably decide to accept the needed efforts,

Hmm.. that works, if you have to touch some (some < 5) boards. But If you have to touch > 5 boards, this gets unhandy...

How about:

Quoted 8 lines
Quoted 3 lines

MAKEALL --check-boards -s at91

;)

I feel this is the hard part of the problem, and what we're glossing over. What has to be tested by the board maintainer? What are we going to leave to their discretion? Will am335x_evm not count if I don't dig up the NOR cape for it?

for the time being I'd glad to see reports of (un)successful boot with configured bootm command.

But I see your point, there is another input vector for the tests. I think this could only be defined on a per board basis.
To pick up your example, I think it is worth to know that one tested the am335x_evm to boot via NAND. At least for the maintainer, so he can skip that and just test the NOR booting.

Best regards

Andreas Bie?mann

#26 lore raw
In reply to Tom Rini

Dear Tom,

Quoted 5 lines · In message <20131107133159.GR5925@bill-the-cat> you wrote:

I feel this is the hard part of the problem, and what we're glossing over. What has to be tested by the board maintainer? What are we going to leave to their discretion? Will am335x_evm not count if I don't dig up the NOR cape for it?

Good question. Eventually this is something that develops over time.

Intially, we might be satisfied with a very basic "it works" message, which may just mean that this specific version booted on the actual hardware.

In the long run, we might provide a more detailed questionaire to the reported. I could for example imagine a tool that parses the board's config file and then provides some checkboxes - if there is NOR flash configured on the board, ask if NOR has been tested; similar for network, MMC, USB, ... other features.

One day we might even have more developers using automatic test tools so we could generate information on a per-command base.

I know that I'm just dreaming, but we should try to just be open for any such future extensions, even if we start really small now.

I think we all agree that _any_ kind of test information will be better than none.

Best regards,

Wolfgang Denk

Signature · 6 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
You see things; and you say ``Why?'' But I dream  things  that  never
were; and I say ``Why not?''
       - George Bernard Shaw _Back to Methuselah_ (1921) pt. 1, act 1
#27 lore raw
Quoted 21 lines · Wolfgang Denk

Dear Tom,

Quoted 5 lines · In message <20131107133159.GR5925@bill-the-cat> you wrote:

I feel this is the hard part of the problem, and what we're glossing over. What has to be tested by the board maintainer? What are we going to leave to their discretion? Will am335x_evm not count if I don't dig up the NOR cape for it?

Good question. Eventually this is something that develops over time.

Intially, we might be satisfied with a very basic "it works" message, which may just mean that this specific version booted on the actual hardware.

In the long run, we might provide a more detailed questionaire to the reported. I could for example imagine a tool that parses the board's config file and then provides some checkboxes - if there is NOR flash configured on the board, ask if NOR has been tested; similar for network, MMC, USB, ... other features.

Quoted 8 lines

One day we might even have more developers using automatic test tools so we could generate information on a per-command base.

I know that I'm just dreaming, but we should try to just be open for any such future extensions, even if we start really small now.

I think we all agree that _any_ kind of test information will be better than none.

What we need to be careful of here is making sure whatever we grow is both useful and not overly complicated. What I honestly wonder about is automated testing for commands (crc32 pops to mind only because I just fixed things) but otherwise having things broken down into a front end where people select what they did "Booted a ___ into ___ via ___", provide some output from a command (maybe add just a touch more info to 'version') and cover non-boot testing with copy/paste'able drop-downs.

I know automated testing is The Thing, but given N frameworks, everyone of them has issues because frankly, every SoC family has its own quirks about how boot and load and what is and is not even feasible, especially for the bootloader.

Signature · 8 lines
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: Digital signature
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20131107/66b988cc/attachment.pgp>
#28 lore raw

Dear Tom,

Quoted 9 lines · In message <20131107205133.GT5925@bill-the-cat> you wrote:

What we need to be careful of here is making sure whatever we grow is both useful and not overly complicated. What I honestly wonder about is automated testing for commands (crc32 pops to mind only because I just fixed things) but otherwise having things broken down into a front end where people select what they did "Booted a ___ into ___ via ___", provide some output from a command (maybe add just a touch more info to 'version') and cover non-boot testing with copy/paste'able drop-downs.

Quoted 4 lines

I know automated testing is The Thing, but given N frameworks, everyone of them has issues because frankly, every SoC family has its own quirks about how boot and load and what is and is not even feasible, especially for the bootloader.

Agreed. In the end, you will probably define a specific set of test cases that can be run automatically on a board. We will never have 100% coverage, nor any generic set of tests that fits all boards.

OK, a pretty large sub-set of common functionality can be tested in the sandbox, which is a great achievement ...

I still firmly belive that the approach we've taken a long time ago, i. e. to use a test framework (DUTS) in combination with a (today wiki based) documentation framework (DULG) is something that can meet a wide range of requirements.

Unfortunately, we haven't been able yet to find a test framework that makes it easy for the average user to add new test cases. I'm not even talking here about the core of the framework, that has to deal with things like attaching to the console of a board, control power and/or reset state, detect if a JTAG debugger is attached and such things. In the current state, our code is expect based (and thus implemented in tcl), and it's really a PITA to add new test cases to it.

BUt it would require significant efforts to rwrite this in a more modern context...

Best regards,

Wolfgang Denk

Signature · 4 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
In C we had to code our own bugs, in C++ we can inherit them.
#29 lore raw
In reply to Wolfgang Denk

Hello Wolfgang,

Am 07.11.2013 20:26, schrieb Wolfgang Denk:

Quoted 10 lines

Dear Tom,

Quoted 5 lines · In message<20131107133159.GR5925@bill-the-cat> you wrote:

I feel this is the hard part of the problem, and what we're glossing over. What has to be tested by the board maintainer? What are we going to leave to their discretion? Will am335x_evm not count if I don't dig up the NOR cape for it?

Good question. Eventually this is something that develops over time.

Yes, let us go step by step.

Quoted 9 lines

Intially, we might be satisfied with a very basic "it works" message, which may just mean that this specific version booted on the actual hardware.

In the long run, we might provide a more detailed questionaire to the reported. I could for example imagine a tool that parses the board's config file and then provides some checkboxes - if there is NOR flash configured on the board, ask if NOR has been tested; similar for network, MMC, USB, ... other features.

Cool... and we can maybe start testscripts based on this information... (Huch... I am sure I am sitting in Front of my PC and not sleeping
anymore)

Quoted 5 lines

One day we might even have more developers using automatic test tools so we could generate information on a per-command base.

I know that I'm just dreaming, but we should try to just be open for any such future extensions, even if we start really small now.

Maybe we add in tools a directory, where we add such tests? But this would be another big discussion...

Quoted 2 lines

I think we all agree that _any_ kind of test information will be better than none.

Yes.

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
#30 lore raw
In reply to Tom Rini

Hello Tom,

Am 07.11.2013 14:31, schrieb Tom Rini:

Quoted 16 lines · Andreas Bie?mann
Quoted 4 lines

Hello all together,

Quoted 1 line · Heiko Schocher

Am 06.11.2013 08:50, schrieb Wolfgang Denk:

[snip]

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

So when you're once again doing some change that requires touching files for some othe rboards, you could simply check that database. If you see that 3 out of the last 5 releases have reported succesful run-time tests you will probably decide to accept the needed efforts,

Hmm.. that works, if you have to touch some (some< 5) boards. But If you have to touch> 5 boards, this gets unhandy...

How about:

Quoted 8 lines
Quoted 3 lines

MAKEALL --check-boards -s at91

;)

I feel this is the hard part of the problem, and what we're glossing over. What has to be tested by the board maintainer? What are we going to leave to their discretion? Will am335x_evm not count if I don't dig up the NOR cape for it?

Good point! What gets tested on a board can only decide the person who execute the test on the board. We must trust, that it covers hopefully all functionality ...

Hmm... would we want to introduce testscripts to U-Boot, which should be executed on boards? Hehe... I see, this discussion breaks my fast thought, of introducing just a small and fast livetimer mechanism ;-)

bye,
Heiko

Signature · 2 lines
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany