BitBake changes in the Yocto Project 1.5 cycle
I've been giving some thought to where BitBake needs to go in the future in order to deliver for its users. It started life as a commandline utility and its grown a lot since it was first created. I think there are some key decisions that need to be taken to ensure its future growth.
The first proposal is that we should change the BitBake server so it becomes memory resident. This means that the first time you run "bitbake X", the server loads into memory, then subsequent BitBake commands would just connect to the server and do things. We'd add in some kind of timeout of say 15 minutes so that it would gracefully exit.
The reason for doing this is simple, it would allow commands to be much more responsive rather than having the cache/configuration loading each time which is where our current overhead is. Obviously it would detect changes to things like MACHINE setting, local.conf and re-parse as normal in those cases. The intent would be to speed up the interaction with the system so you don't have the annoying delays/lag.
This would then lead into the second proposal which is to add better support for new commands. This would be along the lines of some of Chris' work with bb, the bitbake-env script and so on but with the added advantage of being able to connect to the server for info. It would allow us to add query commands for things like the available PACKAGECONFIG options/settings much more easily than the current infrastructure. There are some issues about whether these would belong as part of BitBake or OE and we'd need to figure that out.
The third proposal is to do away with the BitBake wrapper script in OE-Core for pseudo. Doing this within BitBake is going to be hard and require some rework but ultimately, I think it will be worth it from a performance perspective and also stops some of the user confusion about BitBake running twice (two sets of log files and so on).
These changes should also be in keeping with the expanded UI work and options such as WebHob and allowing remote use of multiple UIs connected to servers.
Feedback/comments/suggestions etc./ welcome as always.
I've cc'd the OE-Core list but further discussion should be taken to bitbake-devel.
Cheers,
Richard
Quoted 2 lines · Richard Purdie
We'd add in some kind of
timeout of say 15 minutes so that it would gracefully exit.
Hopefully this time-out will start at the completion of the most recent bitbake command, rather than from its start?
Quoted 3 lines
The reason for doing this is simple, it would allow commands to be much more responsive rather than having the cache/configuration loading each time which is where our current overhead is.
This sounds like a very nice feature to have.
Quoted 2 lines
This would then lead into the second proposal which is to add better support for new commands.
It would be nice if there was a way to zero-in on an existing environment variable. For example:
$ bitbake -e <image>
produces a lot of information which I then have to parse in order to figure out how a given environment variable is being set. It would be nice if I could simply:
$ bitbake -e <image> <variable>
and just get the pertinent parts.
Quoted 5 lines · Trevor Woerner
Quoted 2 lines · Richard Purdie
We'd add in some kind of
timeout of say 15 minutes so that it would gracefully exit.Hopefully this time-out will start at the completion of the most recent bitbake command, rather than from its start?
It would be good if it was configurable. I don't mind leaving and instance running 8 hours for example, so I pay the price once for the project for the day. Also we will want a way to query the status of the instance that is running and kill it. If the instance is periodically checking that its local.conf still exists when it is idle, it can shut itself down if the build directory has been removed. Obviously these things are all in the "nice to have" camp.
Cheers,
Jason.
Quoted 21 lines · Trevor Woerner
Quoted 2 lines · Richard Purdie
We'd add in some kind of
timeout of say 15 minutes so that it would gracefully exit.Hopefully this time-out will start at the completion of the most recent bitbake command, rather than from its start?
Quoted 3 lines
The reason for doing this is simple, it would allow commands to be much more responsive rather than having the cache/configuration loading each time which is where our current overhead is.
This sounds like a very nice feature to have.
Quoted 2 lines
This would then lead into the second proposal which is to add better support for new commands.
It would be nice if there was a way to zero-in on an existing environment variable. For example:
$ bitbake -e <image>
Quoted 7 lines
produces a lot of information which I then have to parse in order to figure out how a given environment variable is being set. It would be nice if I could simply:
$ bitbake -e <image> <variable>and just get the pertinent parts.
i think bitbake-env would do this for you.
rday
Signature · 7 lines
========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca
Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================Quoted 2 lines · Robert P. J. Day
Quoted 1 line · Trevor Woerner
On Mon, Apr 22, 2013 at 10:16 AM, Richard Purdie <[email protected]>
Quoted 12 lines · wrote:
Quoted 12 lines
Quoted 2 lines
This would then lead into the second proposal which is to add better support for new commands.
It would be nice if there was a way to zero-in on an existing environment variable. For example:
$ bitbake -e <image>produces a lot of information which I then have to parse in order to figure out how a given environment variable is being set. It would be nice if I could simply:
Quoted 7 lines
Quoted 3 lines
$ bitbake -e <image> <variable>and just get the pertinent parts.
i think bitbake-env would do this for you.
rday
If I use bitbake-env I just get the current value:
$ bitbake-env OE_TERMINAL
OE_TERMINAL="screen"but if I use "bitbake -e" I get a better pictures of why that variable is what it is
(which I've found very useful):
$ bitbake -e | less
[...snip...]
# $OE_TERMINAL [4 operations]
# set /home/trevor/devel/yocto/builds/fullbuild/conf/local.conf:23
# "screen"
# set?
/home/trevor/devel/yocto/git/poky/meta/classes/terminal.bbclass:1
# "auto"
# set
/home/trevor/devel/yocto/git/poky/meta/classes/terminal.bbclass:2
# [type] "choice"
# set
/home/trevor/devel/yocto/git/poky/meta/classes/terminal.bbclass:5
# [choices] "auto none ${@" ".join(o.name
for o in oe.terminal.prioritized())}"
# computed:
# "screen"
OE_TERMINAL="screen"
[...snip...]W dniu 22.04.2013 16:16, Richard Purdie pisze:
Quoted 5 lines
I've been giving some thought to where BitBake needs to go in the future in order to deliver for its users. It started life as a commandline utility and its grown a lot since it was first created. I think there are some key decisions that need to be taken to ensure its future growth.
There is one thing which kind of bothers me. "ERROR" message which is just a warning. Like:
ERROR: libelf is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libasm is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libdw is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libdw-dev is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libasm-dev is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libelf-dev is listed in PACKAGES multiple times, this leads to packaging errors.
If it is error then let BitBake exit instead of doing task. If it is not error but just a warning then let it be renamed?
My recent build failed with:
--------
NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Started
NOTE: recipe linaro-image-minimal-1.0-r0: task do_rm_work_all: Succeeded
NOTE: Tasks Summary: Attempted 2247 tasks of which 915 didn't need to be rerun and all succeeded.
Summary: There were 3 WARNING messages shown.
Summary: There were 6 ERROR messages shown, returning a non-zero exit code. Build step 'Execute shell' marked build as failure
--------
"all succeeded" does not look like a reason to fail a build.
Quoted 17 lines · Marcin Juszkiewicz
W dniu 22.04.2013 16:16, Richard Purdie pisze:
Quoted 5 lines
I've been giving some thought to where BitBake needs to go in the future in order to deliver for its users. It started life as a commandline utility and its grown a lot since it was first created. I think there are some key decisions that need to be taken to ensure its future growth.
There is one thing which kind of bothers me. "ERROR" message which is just a warning. Like:
ERROR: libelf is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libasm is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libdw is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libdw-dev is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libasm-dev is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libelf-dev is listed in PACKAGES multiple times, this leads to packaging errors.
Quoted 2 lines
If it is error then let BitBake exit instead of doing task. If it is not error but just a warning then let it be renamed?
What we're trying to do is move everything to use a standard mechanism for reporting issues of this type. With insane.bbclass, you can elect whether a given type of error is a warning or errors and fails the task.
There are however several places where bb.error is used without that mechanism. I'd love to see them fixed, please file a bug for it...
Quoted 7 lines
My recent build failed with:
--------
NOTE: recipe linaro-image-minimal-1.0-r0: taskdo_rm_work_all: Started
NOTE: recipe linaro-image-minimal-1.0-r0: taskdo_rm_work_all: Succeeded
NOTE: Tasks Summary: Attempted 2247 tasks of which 915 didn't need to be rerun and all succeeded.
Quoted 6 lines
Summary: There were 3 WARNING messages shown.
Summary: There were 6 ERROR messages shown, returning a non-zero exit code. Build step 'Execute shell' marked build as failure
--------"all succeeded" does not look like a reason to fail a build.
Agreed, please open a bug for it.
FWIW Bitbake will set a non-zero exit code if any error messages appear, as the message says. Whilst people find that confusing, I continue to believe it is the right thing to be doing and we need to tighten up things in other places as I describe above. That error code does help us catch problems on the autobuilders and so forth.
Cheers,
Richard
W dniu 23.04.2013 11:04, Richard Purdie pisze:
Quoted 2 lines
Quoted 1 line
Quoted 1 line
"all succeeded" does not look like a reason to fail a build.
Agreed, please open a bug for it.
Quoted 6 lines · Marcin Juszkiewicz
W dniu 23.04.2013 11:04, Richard Purdie pisze:
Quoted 2 lines
Quoted 1 line
Quoted 1 line
"all succeeded" does not look like a reason to fail a build.
Agreed, please open a bug for it.
I marked 4397 as the duplication of this one:-).
// Robert
Quoted 6 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Hello,
if there was a plan to divide bitbake into a server and a client part, I would suggest to add a kind of "fetching proxy" functionality or run mode to the bitbake sever part.
The reason behind this suggestion is: If you sitting behind a restrictive proxy/firewall and want to use yocto (OE) you always get problems to load from git:// svn:// or cvn:// URLs.
Here it may be useful to be able to run a bb fetch proxy outside of the firewall that tunnels the protocols by https or ssh (via (ssl)port 443 like github does for git), or acts as a flexible download server, do the fetching part and providing the download archives to an other server through a common usable protocol tunnel.
May be this was a special issue, but it may ease the use of yocto (OE) extremely.
Regards
Wolfgang Hauser
Quoted 21 lines · Richard Purdie
Quoted 17 lines · Marcin Juszkiewicz
W dniu 22.04.2013 16:16, Richard Purdie pisze:
Quoted 5 lines
I've been giving some thought to where BitBake needs to go in the future in order to deliver for its users. It started life as a commandline utility and its grown a lot since it was first created. I think there are some key decisions that need to be taken to ensure its future growth.
There is one thing which kind of bothers me. "ERROR" message which is just a warning. Like:
ERROR: libelf is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libasm is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libdw is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libdw-dev is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libasm-dev is listed in PACKAGES multiple times, this leads to packaging errors.
ERROR: libelf-dev is listed in PACKAGES multiple times, this leads to packaging errors.Quoted 2 lines
If it is error then let BitBake exit instead of doing task. If it is not error but just a warning then let it be renamed?
Quoted 7 lines
What we're trying to do is move everything to use a standard mechanism for reporting issues of this type. With insane.bbclass, you can elect whether a given type of error is a warning or errors and fails the task.
There are however several places where bb.error is used without that mechanism. I'd love to see them fixed, please file a bug for it...
Filed a bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4396
Quoted 7 lines
Quoted 7 lines
My recent build failed with:
--------
NOTE: recipe linaro-image-minimal-1.0-r0: taskdo_rm_work_all: Started
NOTE: recipe linaro-image-minimal-1.0-r0: taskdo_rm_work_all: Succeeded
NOTE: Tasks Summary: Attempted 2247 tasks of which 915 didn't need to be rerun and all succeeded.
Quoted 9 lines
Quoted 6 lines
Summary: There were 3 WARNING messages shown.
Summary: There were 6 ERROR messages shown, returning a non-zero exit code. Build step 'Execute shell' marked build as failure
--------"all succeeded" does not look like a reason to fail a build.
Agreed, please open a bug for it.
Filed another bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4398
// Robert
Quoted 20 lines
FWIW Bitbake will set a non-zero exit code if any error messages appear, as the message says. Whilst people find that confusing, I continue to believe it is the right thing to be doing and we need to tighten up things in other places as I describe above. That error code does help us catch problems on the autobuilders and so forth.
Cheers,
Richard
_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
Quoted 22 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Richard Purdie
Sent: den 22 april 2013 16:16
To: bitbake-devel
Cc: openembedded-core
Subject: [OE-core] BitBake changes in the Yocto Project 1.5 cycleI've been giving some thought to where BitBake needs to go in the future in order to deliver for its users. It started life as a commandline utility and its grown a lot since it was first created. I think there are some key decisions that need to be taken to ensure its future growth.
The first proposal is that we should change the BitBake server so it becomes memory resident. This means that the first time you run "bitbake X", the server loads into memory, then subsequent BitBake commands would just connect to the server and do things. We'd add in some kind of timeout of say 15 minutes so that it would gracefully exit.
Quoted 7 lines
The reason for doing this is simple, it would allow commands to be much more responsive rather than having the cache/configuration loading each time which is where our current overhead is. Obviously it would detect changes to things like MACHINE setting, local.conf and re-parse as normal in those cases. The intent would be to speed up the interaction with the system so you don't have the annoying delays/lag.
How will a bitbake command know which server to connect to in case of multiple concurrent builds of different products on the same computer? Or do you intend for one server to keep track of all builds?
[cut]
Quoted 3 lines
These changes should also be in keeping with the expanded UI work and options such as WebHob and allowing remote use of multiple UIs connected to servers.
How should a remote UI know which server to connect to (if there are multiple ones)?
Quoted 8 lines
Feedback/comments/suggestions etc./ welcome as always.
I've cc'd the OE-Core list but further discussion should be taken to bitbake-devel.
Cheers,
Richard
//Peter
Quoted 30 lines · Peter Kjellerstedt
Quoted 22 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Richard Purdie
Sent: den 22 april 2013 16:16
To: bitbake-devel
Cc: openembedded-core
Subject: [OE-core] BitBake changes in the Yocto Project 1.5 cycleI've been giving some thought to where BitBake needs to go in the future in order to deliver for its users. It started life as a commandline utility and its grown a lot since it was first created. I think there are some key decisions that need to be taken to ensure its future growth.
The first proposal is that we should change the BitBake server so it becomes memory resident. This means that the first time you run "bitbake X", the server loads into memory, then subsequent BitBake commands would just connect to the server and do things. We'd add in some kind of timeout of say 15 minutes so that it would gracefully exit.
Quoted 7 lines
The reason for doing this is simple, it would allow commands to be much more responsive rather than having the cache/configuration loading each time which is where our current overhead is. Obviously it would detect changes to things like MACHINE setting, local.conf and re-parse as normal in those cases. The intent would be to speed up the interaction with the system so you don't have the annoying delays/lag.
Quoted 3 lines
How will a bitbake command know which server to connect to in case of multiple concurrent builds of different products on the same computer? Or do you intend for one server to keep track of all builds?
There would continue to be one server per build. I tried to keep away from specifics of the exact implementation since the answer right now is "don't know". I do have ideas, for example we may do this through some kind of file in the build directory like the bitbake.lock file and/or some kind of variable behaving like DISPLAY. I don't really want to reinvent the wheel though and am open to other ideas. Dbus falls down with the "remote" part of what we want to do since its typically only been used locally and whilst people had ideas about remote parts for it, I don't think they were ever developed extensively.
Quoted 6 lines
Quoted 3 lines
These changes should also be in keeping with the expanded UI work and options such as WebHob and allowing remote use of multiple UIs connected to servers.
How should a remote UI know which server to connect to (if there are multiple ones)?
Right now there are commandline options to bitbake for this. This may become some kind of DISPLAY equivalent environment variable, not sure yet.
Cheers,
Richard