Quality of meta-oe metadata
Hi, sorry for longer e-mail, this is one of topic I would like to discuss on OEDAM (http://openembedded.org/wiki/OEDAM), but having some feedback and thoughts in advance will be very useful.
As people can notice from my "State of bitbake world" e-mails or
http://www.openembedded.org/wiki/Bitbake_World_Status
we never had "green" builds. There are always 20+ failed tasks in those big builds and just reading the numbers isn't good indicator of quality, because sooner you break something in dependency tree, fewer recipes will be actually tested, so fewer failed tasks often means that something important is broken.
There are IMHO at least 3 reasons for this depressing state:
1) Nobody is paid/dedicated to fix them, there is no company behind meta-oe layers, now SWAT team, not even dedicated maintainers to which I could send error from latest build and ask them to fix it before the end of week/month.
Kudos to people who are sometimes sending patches for such issues!
2) There are a lot of changes and component upgrades in oe-core which sometimes aren't very straight-forward to adapt to and issues stay in meta-oe for months.
I don't mean it's oe-core fault or that changes to oe-core should slow down just because meta-oe, especially when we cannot guarantee when it will be prepared for them (because of 1)).
oe-core is trying to track latest stable versions, but meta-oe often contains very old versions and people upgrade to latest stable only the recipes they really care about, so it's not so surprising that 2 year old version of something isn't compatible with latest greatest freetype or some library like that.
3) OE releases work great and don't invalidate sstate signatures so often, so my feeling is that most developers and projects are just using releases and less and less people do CI. People will start complaining that something is broken in meta-oe only when they are upgrading their project from 1.5 to 1.6 when 1.6 is released and that could be too late for fixing meta-oe issues.
What I'm trying to do with it:
a) sending those e-mails and updating wiki, so that people can easily find if some build failure is common or something which happens only for them (something like oestats-client.bbclass page was providing in oe-classic) It also includes log of QA issues which are usually easy to fix and great way for new people to learn something about OE.
b) trying to refuse all patches which cause new world issue (or new QA warn/err) - sometimes missed in logs, because it's often "hidden" by some other issue and hard to compare 40 issues from previous build with 38 from current.
Also the issues are often triggered later by changes somewhere else...
c) fixing build/qa issues in recipes I've never used or don't even have hardware to test - just based on assumption that something which builds is better than broken build, even when it can have some issues in runtime.
d) contacting people who added the recipe which is now failing, often without reply for months even when I try it multiple times :/
e) moving to "nonworking" directory to mark it as "known-to-be-broken", last resort for recipes where the fix is complicated and it's not known if someone is actually using it (because it was broken for months and nobody replied).
+ easy to find them, because they are still in repository (instead of git rm + revert when someone fixes it)
- layer index probably doesn't find them, because "nonworking" directory level isn't in BBFILES, so maybe meta-broken or meta-nonworking would be better
? some recipes are "broken" just because their dependency is broken, what
to do with such recipe, I usually just say that in commit message when I'm moving them to "nonworking" with their broken dep.
What can we do better? How to motivate more people to do CI and send fixes? When we get to "green" state it will be easier to quickly spot new issues and easier to fix them, because it will be clear what's causing them.
Hello Martin,
Excellent, excellent post!
Quoted 3 lines · Martin Jansa
2) There are a lot of changes and component upgrades in oe-core which sometimes aren't very straight-forward to adapt to and issues stay in meta-oe for months.
Critical bugfixes aside, I think the current system of unrestrained, perpetual package bumps is an issue. If dylan uses version 1 of package XYZ and dora uses version 5, is there any need for the 3 intervening package bumps which occurred between the release of dylan and dora?
If nothing else, these "irrelevant updates" increase the chance of causing problems ;-)
Quoted 3 lines
3) OE releases work great and don't invalidate sstate signatures so often, so my feeling is that most developers and projects are just using releases and less and less people do CI.
Would users prefer a better-tested more-likely-to-work release containing package versions which were several months old, or is staying on the bleeding edge more important? There is always exponentially more work/cost required to be an early adopter.
Quoted 72 lines · Martin Jansa
Hi, sorry for longer e-mail, this is one of topic I would like to discuss on OEDAM (http://openembedded.org/wiki/OEDAM), but having some feedback and thoughts in advance will be very useful.
As people can notice from my "State of bitbake world" e-mails or
http://www.openembedded.org/wiki/Bitbake_World_Status
we never had "green" builds. There are always 20+ failed tasks in those big builds and just reading the numbers isn't good indicator of quality, because sooner you break something in dependency tree, fewer recipes will be actually tested, so fewer failed tasks often means that something important is broken.There are IMHO at least 3 reasons for this depressing state:
1) Nobody is paid/dedicated to fix them, there is no company behind meta-oe layers, now SWAT team, not even dedicated maintainers to which I could send error from latest build and ask them to fix it before the end of week/month.
Kudos to people who are sometimes sending patches for such issues!
2) There are a lot of changes and component upgrades in oe-core which sometimes aren't very straight-forward to adapt to and issues stay in meta-oe for months.
I don't mean it's oe-core fault or that changes to oe-core should slow down just because meta-oe, especially when we cannot guarantee when it will be prepared for them (because of 1)).
oe-core is trying to track latest stable versions, but meta-oe often contains very old versions and people upgrade to latest stable only the recipes they really care about, so it's not so surprising that 2 year old version of something isn't compatible with latest greatest freetype or some library like that.
3) OE releases work great and don't invalidate sstate signatures so often, so my feeling is that most developers and projects are just using releases and less and less people do CI. People will start complaining that something is broken in meta-oe only when they are upgrading their project from 1.5 to 1.6 when 1.6 is released and that could be too late for fixing meta-oe issues.
What I'm trying to do with it:
a) sending those e-mails and updating wiki, so that people can easily find if some build failure is common or something which happens only for them (something like oestats-client.bbclass page was providing in oe-classic) It also includes log of QA issues which are usually easy to fix and great way for new people to learn something about OE.
b) trying to refuse all patches which cause new world issue (or new QA warn/err) - sometimes missed in logs, because it's often "hidden" by some other issue and hard to compare 40 issues from previous build with 38 from current.
Also the issues are often triggered later by changes somewhere else...
c) fixing build/qa issues in recipes I've never used or don't even have hardware to test - just based on assumption that something which builds is better than broken build, even when it can have some issues in runtime.
d) contacting people who added the recipe which is now failing, often without reply for months even when I try it multiple times :/
e) moving to "nonworking" directory to mark it as "known-to-be-broken", last resort for recipes where the fix is complicated and it's not known if someone is actually using it (because it was broken for months and nobody replied).
+ easy to find them, because they are still in repository (instead of git rm + revert when someone fixes it)
- layer index probably doesn't find them, because "nonworking" directory level isn't in BBFILES, so maybe meta-broken or meta-nonworking would be better
? some recipes are "broken" just because their dependency is broken, what
to do with such recipe, I usually just say that in commit message when I'm moving them to "nonworking" with their broken dep.
Quoted 4 lines
What can we do better? How to motivate more people to do CI and send fixes? When we get to "green" state it will be easier to quickly spot new issues and easier to fix them, because it will be clear what's causing them.
Are you discussing meta-oe alone here or all layers in meta-openembedded?
I've got a few ideas thinking slightly outside the box, so these may or may not be workable:
- It might help to try to split the layers down further and reduce the size of meta-oe (for example, create a meta-python layer for all the python libraries that aren't direct dependencies of something non-python in meta-oe) and then try get new maintainers for these sub-layers so that the workload is spread better.
- We could create a new layer for unstable recipes which are 'use at your own risk'. That may be a good place for recipes which don't work on the jenkins builds but do work for some people and are in use (and so probably don't belong in nonworking). This is similar to the meta-broken or meta-nonworking layer idea but would take slightly more recipes in.
- It may be worth taking some aggressive action in sync with the upcoming oe-core release to get us to a green build, possibly by throwing things into meta-nonworking and meta-unstable layers. That may break a few people's builds, but the fix for them should just be to add the meta-unstable layer. If they're building against the master branch that should be tolerable and it won't affect anyone building from a released/stable branch until the next oe-core release in 6 months time. Once we have a green build it'll be much easier to QA patches and reject those which break the build.
I don't have much time to give to fixing this myself as I'm busy with other projects. I do have idle computer time though so could run an automated build regularly (probably just a script and a cron job rather than buildbot/jenkins/etc). I won't be able to do a world build for every layer for multiple machines, but I should be able to do some subset. I may also be able to commandeer a spare server over the next few weeks. Is there any particular config which would be beneficial to build regularly?
Thanks,
Signature · 4 lines
Paul Barker Email: [email protected] http://www.paulbarker.me.uk
Quoted 1 line · Paul Barker
Are you discussing meta-oe alone here or all layers in meta-openembedded?
basically all of the layers which are included in my world builds, so it includes all layers in meta-openembedded, most from meta-smartphone, meta-browser, meta-qt5..
Quoted 8 lines
I've got a few ideas thinking slightly outside the box, so these may or may not be workable:
- It might help to try to split the layers down further and reduce the size of meta-oe (for example, create a meta-python layer for all the python libraries that aren't direct dependencies of something non-python in meta-oe) and then try get new maintainers for these sub-layers so that the workload is spread better.
Agreed I would be happy to accept patches creating meta-python layer, ideally sent by someone who will volunteer to maintain it.
Quoted 6 lines
- We could create a new layer for unstable recipes which are 'use at your own risk'. That may be a good place for recipes which don't work on the jenkins builds but do work for some people and are in use (and so probably don't belong in nonworking). This is similar to the meta-broken or meta-nonworking layer idea but would take slightly more recipes in.
The difference between meta-broken and meta-unstable from my POV is that, "broken" should be just temporary place and someone should eventually fix it, but if we move stuff to meta-unstable and stop testing it, then I fear that it will became just junkyard.
If some recipe works fine for someone when building for arm, but triggers "jenkins/world" build issues for x86* then lets restrict it only for arm with comment which shows the error - that's better than moving it to "broken" or "unstable".
Quoted 9 lines
- It may be worth taking some aggressive action in sync with the upcoming oe-core release to get us to a green build, possibly by throwing things into meta-nonworking and meta-unstable layers. That may break a few people's builds, but the fix for them should just be to add the meta-unstable layer. If they're building against the master branch that should be tolerable and it won't affect anyone building from a released/stable branch until the next oe-core release in 6 months time. Once we have a green build it'll be much easier to QA patches and reject those which break the build.
you mean being aggressive before or after creating the "next-release" branch? I would prefer to be aggressive before to have green builds in release branch.
Quoted 8 lines
I don't have much time to give to fixing this myself as I'm busy with other projects. I do have idle computer time though so could run an automated build regularly (probably just a script and a cron job rather than buildbot/jenkins/etc). I won't be able to do a world build for every layer for multiple machines, but I should be able to do some subset. I may also be able to commandeer a spare server over the next few weeks. Is there any particular config which would be beneficial to build regularly?
Doing "big" builds in different setups is of course useful, but right now I think we already have "more logs than what we're processing".
So I think that building not whole world, but those recipes which are regularly failing would be good start, if people cannot reproduce some issues which are shown in my jenkins builds we should compare the builds and narrow possible reasons (e.g. failing only with dash, failing only with some PACKAGECONFIG enabled or disabled).
I'm trying to stay close to distroless config, but some tweaks are needed in order to have bigger test coverage (e.g. enabling some PACKAGECONFIGs which are disabled by default, but something requires them, or changing P_V to newer again because something else needs it, enabling gold, because it's more strict so it can catch more issues..)
Jenkins builds are running almost non-stop (because it usually takes around 24 hours per architecture), so often when I want to debug the issue directly on that server in WORKDIR where it failed it's already gone from tmpfs and the workspace is already "blocked" by next build.
Regards,
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 13 lines · Martin Jansa
Quoted 6 lines · Paul Barker
- We could create a new layer for unstable recipes which are 'use at your own risk'. That may be a good place for recipes which don't work on the jenkins builds but do work for some people and are in use (and so probably don't belong in nonworking). This is similar to the meta-broken or meta-nonworking layer idea but would take slightly more recipes in.
The difference between meta-broken and meta-unstable from my POV is that, "broken" should be just temporary place and someone should eventually fix it, but if we move stuff to meta-unstable and stop testing it, then I fear that it will became just junkyard.
Quoted 5 lines
If some recipe works fine for someone when building for arm, but triggers "jenkins/world" build issues for x86* then lets restrict it only for arm with comment which shows the error - that's better than moving it to "broken" or "unstable".
I agree with the worry that this could turn into a junkyard. The problem is, the junk is currently mixed in with the good recipes.
Quoted 10 lines
Quoted 9 lines
- It may be worth taking some aggressive action in sync with the upcoming oe-core release to get us to a green build, possibly by throwing things into meta-nonworking and meta-unstable layers. That may break a few people's builds, but the fix for them should just be to add the meta-unstable layer. If they're building against the master branch that should be tolerable and it won't affect anyone building from a released/stable branch until the next oe-core release in 6 months time. Once we have a green build it'll be much easier to QA patches and reject those which break the build.
Quoted 3 lines
you mean being aggressive before or after creating the "next-release" branch? I would prefer to be aggressive before to have green builds in release branch.
That would probably make more sense.
Quoted 25 lines
Quoted 9 lines
I don't have much time to give to fixing this myself as I'm busy with other projects. I do have idle computer time though so could run an automated build regularly (probably just a script and a cron job rather than buildbot/jenkins/etc). I won't be able to do a world build for every layer for multiple machines, but I should be able to do some subset. I may also be able to commandeer a spare server over the next few weeks. Is there any particular config which would be beneficial to build regularly?
Doing "big" builds in different setups is of course useful, but right now I think we already have "more logs than what we're processing".
So I think that building not whole world, but those recipes which are regularly failing would be good start, if people cannot reproduce some issues which are shown in my jenkins builds we should compare the builds and narrow possible reasons (e.g. failing only with dash, failing only with some PACKAGECONFIG enabled or disabled).
I'm trying to stay close to distroless config, but some tweaks are needed in order to have bigger test coverage (e.g. enabling some PACKAGECONFIGs which are disabled by default, but something requires them, or changing
P_Vto newer again because something else needs it, enabling gold, because it's more strict so it can catch more issues..)
Quoted 5 lines
Jenkins builds are running almost non-stop (because it usually takes around 24 hours per architecture), so often when I want to debug the issue directly on that server in WORKDIR where it failed it's already gone from tmpfs and the workspace is already "blocked" by next build.
Having a more focussed build would help but I can't really give any ability to debug it on the machine I run builds on and I don't really have time to debug much myself. It would literally just be a status and a set of logs for a different config.
I think we need to prioritise what needs fixing first. I think doing a build of meta-oe only with no PACKAGECONFIG changes, disabling anything that requires PACKAGECONFIG changes, for qemuarm and qemux86 (and qemux86_64 if I have time) would be a good start to see what fails in that case. Then step out to further layers once meta-oe is green.
Signature · 4 lines
Paul Barker Email: [email protected] http://www.paulbarker.me.uk
Quoted 6 lines · Martin Jansa
There are always 20+ failed tasks in those
big builds and just reading the numbers isn't good indicator of quality, because sooner you break something in dependency tree, fewer recipes will be actually tested, so fewer failed tasks often means that something important is broken.
Sorry to double reply but had a thought on this one thing in particular.
Is there any way to list recipes (or tasks) which weren't executed because a dependency failed? I know we see this at the start of the build if dependencies can't be satisfied, I'm thinking tasks with satisfied dependencies but then those dependencies fail.
Being able to see that chain for each build may help us prioritise. It would also show whether a failure disappearing was due to the issue being fixed or the recipe no longer being attempted due to a new failure somewhere else.
Thanks,
Signature · 4 lines
Paul Barker Email: [email protected] http://www.paulbarker.me.uk
Quoted 5 lines · Trevor Woerner
Hello Martin,
Excellent, excellent post!
On 03/29/14 21:31, Martin Jansa wrote:
Quoted 8 lines
Quoted 3 lines
2) There are a lot of changes and component upgrades in oe-core which sometimes aren't very straight-forward to adapt to and issues stay in meta-oe for months.
Critical bugfixes aside, I think the current system of unrestrained, perpetual package bumps is an issue. If dylan uses version 1 of package XYZ and dora uses version 5, is there any need for the 3 intervening package bumps which occurred between the release of dylan and dora?
To put the other side of the argument to this, if you leave things and just update once, you bring in more change and it usually ends up being progressively harder to debug any issues since more things changed and you don't know which change caused which problem.
OE-Core has there fore gone for the "update regularly" philospohy where we can. This means we get to know about issues earlier and have more time to fix them, they're also less likely to get confused with other bugs.
Quoted 2 lines
If nothing else, these "irrelevant updates" increase the chance of causing problems ;-)
I'd disagree :)
Quoted 8 lines
Quoted 3 lines
3) OE releases work great and don't invalidate sstate signatures so often, so my feeling is that most developers and projects are just using releases and less and less people do CI.
Would users prefer a better-tested more-likely-to-work release containing package versions which were several months old, or is staying on the bleeding edge more important? There is always exponentially more work/cost required to be an early adopter.
At least for OE-Core, there is a strong pressure to try and keep up to date. The stability is found in the release branches one a set of versions are locked in.
If we run into problems and talk to upstreams, the invariable question was "does this reproduce with the last release (or head of SCM)?". If we're up to date, it makes it easier for us to talk to upstreams in that regard too.
Cheers,
Richard
Quoted 11 lines · Martin Jansa
Hi, sorry for longer e-mail, this is one of topic I would like to discuss on OEDAM (http://openembedded.org/wiki/OEDAM), but having some feedback and thoughts in advance will be very useful.
As people can notice from my "State of bitbake world" e-mails or
http://www.openembedded.org/wiki/Bitbake_World_Status
we never had "green" builds. There are always 20+ failed tasks in those big builds and just reading the numbers isn't good indicator of quality, because sooner you break something in dependency tree, fewer recipes will be actually tested, so fewer failed tasks often means that something important is broken.
...
Quoted 6 lines
3) OE releases work great and don't invalidate sstate signatures so often, so my feeling is that most developers and projects are just using releases and less and less people do CI. People will start complaining that something is broken in meta-oe only when they are upgrading their project from 1.5 to 1.6 when 1.6 is released and that could be too late for fixing meta-oe issues.
I agree, the success of what we're doing is certainly causing us 'different' problems. :)
Quoted 17 lines
What I'm trying to do with it:
a) sending those e-mails and updating wiki, so that people can easily find if some build failure is common or something which happens only for them (something like oestats-client.bbclass page was providing in oe-classic) It also includes log of QA issues which are usually easy to fix and great way for new people to learn something about OE.
b) trying to refuse all patches which cause new world issue (or new QA warn/err) - sometimes missed in logs, because it's often "hidden" by some other issue and hard to compare 40 issues from previous build with 38 from current.
Also the issues are often triggered later by changes somewhere else...
c) fixing build/qa issues in recipes I've never used or don't even have hardware to test - just based on assumption that something which builds is better than broken build, even when it can have some issues in runtime.
d) contacting people who added the recipe which is now failing, often without reply for months even when I try it multiple times :/
I agree with all of the above. In fact I suspect you are going above and beyond what you really need to. Kudos for that BTW.
Quoted 4 lines
e) moving to "nonworking" directory to mark it as "known-to-be-broken", last resort for recipes where the fix is complicated and it's not known if someone is actually using it (because it was broken for months and nobody replied).
Quoted 8 lines
+ easy to find them, because they are still in repository (instead of git rm + revert when someone fixes it)
- layer index probably doesn't find them, because "nonworking" directory level isn't in BBFILES, so maybe meta-broken or meta-nonworking would be better
? some recipes are "broken" just because their dependency is broken, what
to do with such recipe, I usually just say that in commit message when I'm moving them to "nonworking" with their broken dep.
Have you considered using the blacklist system for this?
You could do something like:
conf/layer.conf:
include ${LAYERDIR}/conf/broken.inc
conf/broken.inc:
<can we ensure the blacklist system is in the system>
BROKENMSG_layername = "The recipe is disabled due to a build failure. If you need this recipe, or have gotten it to work. Please submit patches to <path>. Otherwise this recipe will be removed in the future."
# Recipe FOO is broken as of 2014-03-14, see ...
PNBLACKLIST[FOO] = "${BROKENMSG_layername}"
# Recipe BAR is broken as of 2013-06-13, see ...
PNBLACKLIST[BAR] = "${BROKENMSG_layername}"
Then after a given amount of time, say one year? on the broken list -- we can then remove the items.
If the format of the comments is such that it can be easily parsed, then we can even automate tracking of these things.
(In cases where dependencies are causing the breakage, the message cause be augmented with that information as well...)
The advantage of the blacklist system is that if a user tries to use the recipe they will hopefully see the blacklist message, it prevents having to git mv recipes, and should be easier for people to find/fix the bad code via a simple patch. (And hopefully easier to remove old cruft!)
--Mark
Quoted 6 lines
What can we do better? How to motivate more people to do CI and send fixes? When we get to "green" state it will be easier to quickly spot new issues and easier to fix them, because it will be clear what's causing them.
Quoted 87 lines · Mark Hatle
Quoted 11 lines · Martin Jansa
Hi, sorry for longer e-mail, this is one of topic I would like to discuss on OEDAM (http://openembedded.org/wiki/OEDAM), but having some feedback and thoughts in advance will be very useful.
As people can notice from my "State of bitbake world" e-mails or
http://www.openembedded.org/wiki/Bitbake_World_Status
we never had "green" builds. There are always 20+ failed tasks in those big builds and just reading the numbers isn't good indicator of quality, because sooner you break something in dependency tree, fewer recipes will be actually tested, so fewer failed tasks often means that something important is broken....
Quoted 6 lines
3) OE releases work great and don't invalidate sstate signatures so often, so my feeling is that most developers and projects are just using releases and less and less people do CI. People will start complaining that something is broken in meta-oe only when they are upgrading their project from 1.5 to 1.6 when 1.6 is released and that could be too late for fixing meta-oe issues.
I agree, the success of what we're doing is certainly causing us 'different' problems. :)
Quoted 17 lines
What I'm trying to do with it:
a) sending those e-mails and updating wiki, so that people can easily find if some build failure is common or something which happens only for them (something like oestats-client.bbclass page was providing in oe-classic) It also includes log of QA issues which are usually easy to fix and great way for new people to learn something about OE.
b) trying to refuse all patches which cause new world issue (or new QA warn/err) - sometimes missed in logs, because it's often "hidden" by some other issue and hard to compare 40 issues from previous build with 38 from current.
Also the issues are often triggered later by changes somewhere else...
c) fixing build/qa issues in recipes I've never used or don't even have hardware to test - just based on assumption that something which builds is better than broken build, even when it can have some issues in runtime.
d) contacting people who added the recipe which is now failing, often without reply for months even when I try it multiple times :/I agree with all of the above. In fact I suspect you are going above and beyond what you really need to. Kudos for that BTW.
Quoted 4 lines
e) moving to "nonworking" directory to mark it as "known-to-be-broken", last resort for recipes where the fix is complicated and it's not known if someone is actually using it (because it was broken for months and nobody replied).
Quoted 8 lines
+ easy to find them, because they are still in repository (instead of git rm + revert when someone fixes it)
- layer index probably doesn't find them, because "nonworking" directory level isn't in BBFILES, so maybe meta-broken or meta-nonworking would be better
? some recipes are "broken" just because their dependency is broken, what
to do with such recipe, I usually just say that in commit message when I'm moving them to "nonworking" with their broken dep.Have you considered using the blacklist system for this?
You could do something like:
conf/layer.conf:
include ${LAYERDIR}/conf/broken.incconf/broken.inc:
<can we ensure the blacklist system is in the system>
BROKENMSG_layername= "The recipe is disabled due to a build failure. If you need this recipe, or have gotten it to work. Please submit patches to <path>. Otherwise this recipe will be removed in the future."# Recipe FOO is broken as of 2014-03-14, see ...
PNBLACKLIST[FOO] = "${BROKENMSG_layername}"# Recipe BAR is broken as of 2013-06-13, see ...
PNBLACKLIST[BAR] = "${BROKENMSG_layername}"Then after a given amount of time, say one year? on the broken list -- we can then remove the items.
If the format of the comments is such that it can be easily parsed, then we can even automate tracking of these things.
Quoted 7 lines
(In cases where dependencies are causing the breakage, the message cause be augmented with that information as well...)
The advantage of the blacklist system is that if a user tries to use the recipe they will hopefully see the blacklist message, it prevents having to git mv recipes, and should be easier for people to find/fix the bad code via a simple patch. (And hopefully easier to remove old cruft!)
Yes, that's another way of doing that and I was using it on world builds as well (but without including it in layer and layer.conf to make it "public")
It definitely has the advantage that you can "document" it in the message and few more details in the file itself.
Disadvantage from my POV was that I never included and enabled it in repo, so new people didn't know about it and will still see the issues when they try to build something broken.
Another disadvantage was that I always felt, OK I'll mark this as broken with PNBLACKLIST and lets forget that it ever existed (sometimes I've uncommented include lines for this just to confirm that everything still fails - but not so often as "regular" builds).
And last one: if I recall correctly, when I was using this it was hard to unblacklist something in your config, so if you wanted to test newer version or something you had to modify world_mask.inc first, which won't be very good for people if we include it by default.
Regards,
Quoted 14 lines
--Mark
Quoted 6 lines
What can we do better? How to motivate more people to do CI and send fixes? When we get to "green" state it will be easier to quickly spot new issues and easier to fix them, because it will be clear what's causing them.
Signature · 4 lines
_______________________________________________ Openembedded-core mailing list [email protected] http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 106 lines · Martin Jansa
Quoted 87 lines · Mark Hatle
Quoted 11 lines · Martin Jansa
Hi, sorry for longer e-mail, this is one of topic I would like to discuss on OEDAM (http://openembedded.org/wiki/OEDAM), but having some feedback and thoughts in advance will be very useful.
As people can notice from my "State of bitbake world" e-mails or
http://www.openembedded.org/wiki/Bitbake_World_Status
we never had "green" builds. There are always 20+ failed tasks in those big builds and just reading the numbers isn't good indicator of quality, because sooner you break something in dependency tree, fewer recipes will be actually tested, so fewer failed tasks often means that something important is broken....
Quoted 6 lines
3) OE releases work great and don't invalidate sstate signatures so often, so my feeling is that most developers and projects are just using releases and less and less people do CI. People will start complaining that something is broken in meta-oe only when they are upgrading their project from 1.5 to 1.6 when 1.6 is released and that could be too late for fixing meta-oe issues.
I agree, the success of what we're doing is certainly causing us 'different' problems. :)
Quoted 17 lines
What I'm trying to do with it:
a) sending those e-mails and updating wiki, so that people can easily find if some build failure is common or something which happens only for them (something like oestats-client.bbclass page was providing in oe-classic) It also includes log of QA issues which are usually easy to fix and great way for new people to learn something about OE.
b) trying to refuse all patches which cause new world issue (or new QA warn/err) - sometimes missed in logs, because it's often "hidden" by some other issue and hard to compare 40 issues from previous build with 38 from current.
Also the issues are often triggered later by changes somewhere else...
c) fixing build/qa issues in recipes I've never used or don't even have hardware to test - just based on assumption that something which builds is better than broken build, even when it can have some issues in runtime.
d) contacting people who added the recipe which is now failing, often without reply for months even when I try it multiple times :/I agree with all of the above. In fact I suspect you are going above and beyond what you really need to. Kudos for that BTW.
Quoted 4 lines
e) moving to "nonworking" directory to mark it as "known-to-be-broken", last resort for recipes where the fix is complicated and it's not known if someone is actually using it (because it was broken for months and nobody replied).
Quoted 8 lines
+ easy to find them, because they are still in repository (instead of git rm + revert when someone fixes it)
- layer index probably doesn't find them, because "nonworking" directory level isn't in BBFILES, so maybe meta-broken or meta-nonworking would be better
? some recipes are "broken" just because their dependency is broken, what
to do with such recipe, I usually just say that in commit message when I'm moving them to "nonworking" with their broken dep.Have you considered using the blacklist system for this?
You could do something like:
conf/layer.conf:
include ${LAYERDIR}/conf/broken.incconf/broken.inc:
<can we ensure the blacklist system is in the system>
BROKENMSG_layername= "The recipe is disabled due to a build failure. If you need this recipe, or have gotten it to work. Please submit patches to <path>. Otherwise this recipe will be removed in the future."# Recipe FOO is broken as of 2014-03-14, see ...
PNBLACKLIST[FOO] = "${BROKENMSG_layername}"# Recipe BAR is broken as of 2013-06-13, see ...
PNBLACKLIST[BAR] = "${BROKENMSG_layername}"Then after a given amount of time, say one year? on the broken list -- we can then remove the items.
If the format of the comments is such that it can be easily parsed, then we can even automate tracking of these things.
Quoted 7 lines
(In cases where dependencies are causing the breakage, the message cause be augmented with that information as well...)
The advantage of the blacklist system is that if a user tries to use the recipe they will hopefully see the blacklist message, it prevents having to git mv recipes, and should be easier for people to find/fix the bad code via a simple patch. (And hopefully easier to remove old cruft!)
Yes, that's another way of doing that and I was using it on world builds as well (but without including it in layer and layer.conf to make it "public")
It definitely has the advantage that you can "document" it in the message and few more details in the file itself.
Quoted 8 lines
Disadvantage from my POV was that I never included and enabled it in repo, so new people didn't know about it and will still see the issues when they try to build something broken.
Another disadvantage was that I always felt, OK I'll mark this as broken with PNBLACKLIST and lets forget that it ever existed (sometimes I've uncommented include lines for this just to confirm that everything still fails - but not so often as "regular" builds).
Ya, it definitely prevents retry without a conscious change to a file.
Quoted 4 lines
And last one: if I recall correctly, when I was using this it was hard to unblacklist something in your config, so if you wanted to test newer version or something you had to modify world_mask.inc first, which won't be very good for people if we include it by default.
To unblacklist, you would set PNBLACKIST[FOO] = ""
But of course, someone has to know then setting it to blank has the effect of removing the blacklist for their work.
--Mark
Quoted 17 lines
Regards,
Quoted 14 lines
--Mark
Quoted 6 lines
What can we do better? How to motivate more people to do CI and send fixes? When we get to "green" state it will be easier to quickly spot new issues and easier to fix them, because it will be clear what's causing them.
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core