Skip to contentSkip to conversation

FAQ entry about why no binary packages

PATCH Superseded Yann E. MORIN 5 participants 8 messages Feb 17, 2014 – Feb 18, 2014

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover FAQ entry about why no binary packages

From: "Yann E. MORIN" <[email protected]>

Hello All!

Every now and then, someone will ask why Buildroot has no support for generating binary packages.

Rather than re-hash the same answer every time, the following patch adds a section in the FAQ in the manual, that we can point users to.

The patch is two-fold:
- first, a (slightly tweaked) copy-paste of the report of a Developpers Day we had in Pragues, in 2011,
- second, a position statement about the issue.

The position statement might be a bit strong in its wordings, so I would very much appreciate comments about that.

Changes v1 -> v2:
- copy the report from the DevDay (Thomas P)

Regards,
Yann E. MORIN.

----------------------------------------------------------------
Yann E. MORIN (1):
manual/faq: add section about why no binary packages

 docs/manual/faq-troubleshooting.txt | 91 +++++++++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#1 lore raw

PATCH v2 1/1 manual/faq: add section about why no binary packages

From: "Yann E. MORIN" <[email protected]>

It comes up every now and then on the list, so better be prepared to point at the manual, rather than rehash the same every time.

Most of the chapter is a copy-paste of the report from the Buildroot Developpers Day in Pragues, 2011-10-28:

    http://lists.busybox.net/pipermail/buildroot/2011-November/047229.html

We consider the opinions expressed in that report to be still valid now, two years later.

Cc Samuel Martin <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Baruch Siach <[email protected]>
---
 docs/manual/faq-troubleshooting.txt | 91 +++++++++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)
docs/manual/faq-troubleshooting.txt +91 -0
index 4e0612b..636b3b6 100644
@@ -111,3 +111,94 @@ directory as the new root, will most likely fail.
 If you want to run the target filesystem inside a chroot, or as an NFS
 root, then use the tarball image generated in +images/+ and extract it
 as root.
+
+[[faq-no-binary-packages]]
+Why doesn't Buildroot generate binary packages (.deb, .ipkg...)?
+----------------------------------------------------------------
+
+One feature that is often discussed on the Buildroot list, is the
+the general topic of "package management". To summarize, the idea
+would be to add some tracking of which Buildroot package installs
+what files, with the goals of:
+
+ * Being able to remove files installed by a package when this package
+   gets unselected from the menuconfig ;
+
+ * Ultimately, be able to generate binary packages (ipk or other
+   format) that can be installed on the target without re-generating a
+   new root filesystem image.
+
+In general, most people think it is easy to do: just track which package
+installed what and remove it when the package is unselected. However, it
+is much more complicated than that:
+
+ * It is not only about the +target/+ directory, but also the sysroot in
+   +host/usr/<tuple>/sysroot+ and the +host/+ directory itself. All files
+   installed in those directories by various packages must be tracked.
+
+ * When a package is removed, it is not sufficient to remove just the
+   files it installed. One must also remove all its reverse
+   dependencies (i.e packages relying on it) and rebuild all those
+   packages. For example, package A depends optionally on the OpenSSL
+   library. Both are selected, and Buildroot is built. Package A is
+   built with crypto support using OpenSSL. Later on, OpenSSL gets
+   unselected from the configuration, but package A remains (since
+   OpenSSL is an optional dependency, this is possible). If you just
+   remove the OpenSSL files, then the files installed by package A are
+   broken: they use a library that is no longer present on the
+   target. Technically, it is possible to do this (the prototype that
+   Lionel Landwerlin and Thomas Petazzoni have worked on started to do
+   this), but it is difficult and adds a fair bit of complexity.
+
+ * In addition to the previous problem, there is the case where the
+   optional dependency is not even known to Buildroot. For example,
+   package A in version 1.0 never used OpenSSL, but in version 2.0 it
+   automatically uses OpenSSL if available. If the Buildroot .mk file
+   hasn't been updated to take this into account, then package A will
+   not be part of the reverse dependencies of OpenSSL and will not be
+   removed and rebuilt when OpenSSL is removed. For sure, the .mk file
+   of package A should be fixed to mention this optional dependency,
+   but in the mean time, you can have non-reproducible behaviors.
+
+ * The whole idea is also to allow changes in the menuconfig to be
+   applied on the output directory without having to rebuild
+   everything from scratch. However, this is very difficult to achieve
+   in a reliable way: what happens when the suboptions of a package
+   are changed (we would have to detect this, and rebuild the package
+   from scratch and potentially all its reverse dependencies), what
+   happens if toolchain options are changed, etc. At the moment, what
+   Buildroot does is clear and simple so its behaviour is very
+   reliable and it is easy to support users. If we start telling users
+   that the configuration changes done in menuconfig are applied after
+   the next make, then it has to work correctly and properly in all
+   situations, and not have some bizarre corner cases. We fear bug
+   reports like "I have enabled package A, B and C, then ran make,
+   then disabled package C and enabled package D and ran make, then
+   re-enabled package C and enabled package E and then there is a
+   build failure". Or worse "I did some configuration, then built,
+   then did some changes, built, some more changes, built, some more
+   changes, built, and now it fails, but I don't remember all the
+   changes I did and in which order". This will be impossible to
+   support.
+
+For all these reasons, the conclusion is that adding tracking of
+installed files to remove them when the package is unselected, or to
+generate a repository of binary packages, is something that is very
+hard to achieve reliably and will add a lot of complexity.
+
+On this matter, the Buildroot developpers make these position statements:
+
+ * Buildroot strives at making it easy to generate a root filesystem
+   (hence the name, by the way). That is what we want to make Buildroot
+   good at: building root filesystems.
+
+ * Buildroot is not meant to be a distribution (or rather, a distribution
+   generator). It is the opinion of most Buildroot developers that this
+   is not a goal we should pursue. We believe that there are other tools
+   better suited to generate a distro than Buildroot is. For example,
+   http://openembedded.org/[Open Embedded], or https://openwrt.org/[openWRT],
+   are such tools.
+
+ * We prefer to push Buildroot in a direction that makes it easy (or even
+   easier) to generate complete root filesystems. This is what makes
+   Buildroot stands out in the crowd (among other things, of course!).
Signature · 1 line
1.8.1.2
#2 lore raw

PATCH 1/1 manual/faq: add section about why no binary packages

Yann,

Nit-picky writing editorial - no issue with the actual content ...

Quoted 28 lines · Yann E. MORIN

From: "Yann E. MORIN" <[email protected]>

It comes up every now and then on the list, so better be prepared to point at the manual, rather than rehash the same every time.

Most of the chapter is a copy-paste of the report from the Buildroot Developpers Day in Pragues, 2011-10-28:

    http://lists.busybox.net/pipermail/buildroot/2011-November/047229.html

We consider the opinions expressed in that report to be still valid now, two years later.

Cc Samuel Martin <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Baruch Siach <[email protected]>
---
 docs/manual/faq-troubleshooting.txt | 91 +++++++++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)
docs/manual/faq-troubleshooting.txt +0 -0
index 4e0612b..636b3b6 100644

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 119

Quoted 9 lines
docs/manual/faq-troubleshooting.txt +6 -0
@@ -111,3 +111,94 @@ directory as the new root, will most likely fail.
 If you want to run the target filesystem inside a chroot, or as an NFS
 root, then use the tarball image generated in +images/+ and extract it
 as root.
+
+[[faq-no-binary-packages]]
+Why doesn't Buildroot generate binary packages (.deb, .ipkg...)?
+----------------------------------------------------------------
+
+One feature that is often discussed on the Buildroot list, is the

no comma

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 125

Quoted 6 lines
docs/manual/faq-troubleshooting.txt +6 -0
+the general topic of "package management". To summarize, the idea
+would be to add some tracking of which Buildroot package installs
+what files, with the goals of:
+
+ * Being able to remove files installed by a package when this package
+   gets unselected from the menuconfig ;

; -> .

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 129

Quoted 4 lines
docs/manual/faq-troubleshooting.txt +4 -0
+
+ * Ultimately, be able to generate binary packages (ipk or other
+   format) that can be installed on the target without re-generating a
+   new root filesystem image.

There's some tense inconsistency here between the first point ("Being able to...") and the second ("be able to...")

Quoted 6 lines
docs/manual/faq-troubleshooting.txt +6 -0
+
+In general, most people think it is easy to do: just track which package
+installed what and remove it when the package is unselected. However, it
+is much more complicated than that:
+
+ * It is not only about the +target/+ directory, but also the sysroot in

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 145

Quoted 10 lines
docs/manual/faq-troubleshooting.txt +10 -0
+   +host/usr/<tuple>/sysroot+ and the +host/+ directory itself. All files
+   installed in those directories by various packages must be tracked.
+
+ * When a package is removed, it is not sufficient to remove just the
+   files it installed. One must also remove all its reverse
+   dependencies (i.e packages relying on it) and rebuild all those
+   packages. For example, package A depends optionally on the OpenSSL
+   library. Both are selected, and Buildroot is built. Package A is
+   built with crypto support using OpenSSL. Later on, OpenSSL gets
+   unselected from the configuration, but package A remains (since

remains (since ,,, possible). -> remains. (Since ... possible.)

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 151

Quoted 6 lines
docs/manual/faq-troubleshooting.txt +6 -0
+   OpenSSL is an optional dependency, this is possible). If you just
+   remove the OpenSSL files, then the files installed by package A are
+   broken: they use a library that is no longer present on the
+   target. Technically, it is possible to do this (the prototype that
+   Lionel Landwerlin and Thomas Petazzoni have worked on started to do
+   this), but it is difficult and adds a fair bit of complexity.

"a fair bit of" may be colloquial. Maybe change to "a lot of"?

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 163

Quoted 12 lines
docs/manual/faq-troubleshooting.txt +12 -0
+
+ * In addition to the previous problem, there is the case where the
+   optional dependency is not even known to Buildroot. For example,
+   package A in version 1.0 never used OpenSSL, but in version 2.0 it
+   automatically uses OpenSSL if available. If the Buildroot .mk file
+   hasn't been updated to take this into account, then package A will
+   not be part of the reverse dependencies of OpenSSL and will not be
+   removed and rebuilt when OpenSSL is removed. For sure, the .mk file
+   of package A should be fixed to mention this optional dependency,
+   but in the mean time, you can have non-reproducible behaviors.
+
+ * The whole idea is also to allow changes in the menuconfig to be

Too conversational: "whole idea". Maybe "idea" or "implied goal"?

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 172

Quoted 9 lines
docs/manual/faq-troubleshooting.txt +9 -0
+   applied on the output directory without having to rebuild
+   everything from scratch. However, this is very difficult to achieve
+   in a reliable way: what happens when the suboptions of a package
+   are changed (we would have to detect this, and rebuild the package
+   from scratch and potentially all its reverse dependencies), what
+   happens if toolchain options are changed, etc. At the moment, what
+   Buildroot does is clear and simple so its behaviour is very
+   reliable and it is easy to support users. If we start telling users
+   that the configuration changes done in menuconfig are applied after

"If we start telling users that" may be too conversational. Maybe "If we claim that"?

Quoted 7 lines
docs/manual/faq-troubleshooting.txt +7 -0
+   the next make, then it has to work correctly and properly in all
+   situations, and not have some bizarre corner cases. We fear bug
+   reports like "I have enabled package A, B and C, then ran make,
+   then disabled package C and enabled package D and ran make, then
+   re-enabled package C and enabled package E and then there is a
+   build failure". Or worse "I did some configuration, then built,
+   then did some changes, built, some more changes, built, some more

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 189

Quoted 10 lines
docs/manual/faq-troubleshooting.txt +10 -0
+   changes, built, and now it fails, but I don't remember all the
+   changes I did and in which order". This will be impossible to
+   support.
+
+For all these reasons, the conclusion is that adding tracking of
+installed files to remove them when the package is unselected, or to
+generate a repository of binary packages, is something that is very
+hard to achieve reliably and will add a lot of complexity.
+
+On this matter, the Buildroot developpers make these position statements:

developpers -> developers

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 191

Quoted 2 lines
docs/manual/faq-troubleshooting.txt +2 -0
+
+ * Buildroot strives at making it easy to generate a root filesystem

strives at making -> strives to make

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 198

Quoted 7 lines
docs/manual/faq-troubleshooting.txt +7 -0
+   (hence the name, by the way). That is what we want to make Buildroot
+   good at: building root filesystems.
+
+ * Buildroot is not meant to be a distribution (or rather, a distribution
+   generator). It is the opinion of most Buildroot developers that this
+   is not a goal we should pursue. We believe that there are other tools
+   better suited to generate a distro than Buildroot is. For example,

than Buildroot is. -> than Buildroot.

Danomi -

Quoted 13 lines
docs/manual/faq-troubleshooting.txt +6 -1
+   http://openembedded.org/[Open Embedded], or https://openwrt.org/[openWRT],
+   are such tools.
+
+ * We prefer to push Buildroot in a direction that makes it easy (or even
+   easier) to generate complete root filesystems. This is what makes
+   Buildroot stands out in the crowd (among other things, of course!).
--

1.8.1.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

#3 lore raw

PATCH v2 1/1 manual/faq: add section about why no binary packages

Danomi, All,

On 2014-02-17 15:21 -0500, Danomi Manchego spake thusly:

Quoted 1 line

Nit-picky writing editorial - no issue with the actual content ...

Thank you!

My answers below...

Quoted 1 line · Yann E. MORIN

[--SNIP--]

Quoted 7 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 119

Quoted 5 lines

+[[faq-no-binary-packages]]

docs/manual/faq-troubleshooting.txt +4 -0
+Why doesn't Buildroot generate binary packages (.deb, .ipkg...)?
+----------------------------------------------------------------
+
+One feature that is often discussed on the Buildroot list, is the

no comma

Fixed.

Quoted 8 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 125

Quoted 6 lines
docs/manual/faq-troubleshooting.txt +6 -0
+the general topic of "package management". To summarize, the idea
+would be to add some tracking of which Buildroot package installs
+what files, with the goals of:
+
+ * Being able to remove files installed by a package when this package
+   gets unselected from the menuconfig ;

; -> .

I am not so sure: this is a list, and to me the second bullet-point is included in that sentence.

Quoted 6 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 129

Quoted 3 lines

+ * Ultimately, be able to generate binary packages (ipk or other

docs/manual/faq-troubleshooting.txt +2 -0
+   format) that can be installed on the target without re-generating a
+   new root filesystem image.

There's some tense inconsistency here between the first point ("Being able to...") and the second ("be able to...")

Fixed.

Quoted 9 lines
Quoted 9 lines
docs/manual/faq-troubleshooting.txt +9 -0
+In general, most people think it is easy to do: just track which package
+installed what and remove it when the package is unselected. However, it
+is much more complicated than that:
+
+ * It is not only about the +target/+ directory, but also the sysroot in
+   +host/usr/<tuple>/sysroot+ and the +host/+ directory itself. All files
+   installed in those directories by various packages must be tracked.
+
+ * When a package is removed, it is not sufficient to remove just the
Quoted 8 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 145

Quoted 6 lines
docs/manual/faq-troubleshooting.txt +6 -0
+   files it installed. One must also remove all its reverse
+   dependencies (i.e packages relying on it) and rebuild all those
+   packages. For example, package A depends optionally on the OpenSSL
+   library. Both are selected, and Buildroot is built. Package A is
+   built with crypto support using OpenSSL. Later on, OpenSSL gets
+   unselected from the configuration, but package A remains (since

remains (since ,,, possible). -> remains. (Since ... possible.)

The phrase between parentheses is part of the previous sentence, so no upper-case is needed for 'since'.

However, the english grammar rules dictate that the terminating punctuation be inside the parentheses, not outside. Fixed.

Quoted 8 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 151

Quoted 6 lines
docs/manual/faq-troubleshooting.txt +6 -0
+   OpenSSL is an optional dependency, this is possible). If you just
+   remove the OpenSSL files, then the files installed by package A are
+   broken: they use a library that is no longer present on the
+   target. Technically, it is possible to do this (the prototype that
+   Lionel Landwerlin and Thomas Petazzoni have worked on started to do
+   this), but it is difficult and adds a fair bit of complexity.

"a fair bit of" may be colloquial. Maybe change to "a lot of"?

Well, I like the turn of phrase, as it emphasises the fact that it is not easy (see eg. http://www.merriam-webster.com/dictionary/fair
defintion 10b: moderately numerous, large, or significant).

Quoted 12 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 163

Quoted 11 lines

+ * In addition to the previous problem, there is the case where the

docs/manual/faq-troubleshooting.txt +10 -0
+   optional dependency is not even known to Buildroot. For example,
+   package A in version 1.0 never used OpenSSL, but in version 2.0 it
+   automatically uses OpenSSL if available. If the Buildroot .mk file
+   hasn't been updated to take this into account, then package A will
+   not be part of the reverse dependencies of OpenSSL and will not be
+   removed and rebuilt when OpenSSL is removed. For sure, the .mk file
+   of package A should be fixed to mention this optional dependency,
+   but in the mean time, you can have non-reproducible behaviors.
+
+ * The whole idea is also to allow changes in the menuconfig to be
Quoted 1 line

Too conversational: "whole idea". Maybe "idea" or "implied goal"?

I've rephrased that to read:
The request is to also allow changes in the menuconfig [...]

Quoted 10 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 172

Quoted 9 lines
docs/manual/faq-troubleshooting.txt +9 -0
+   applied on the output directory without having to rebuild
+   everything from scratch. However, this is very difficult to achieve
+   in a reliable way: what happens when the suboptions of a package
+   are changed (we would have to detect this, and rebuild the package
+   from scratch and potentially all its reverse dependencies), what
+   happens if toolchain options are changed, etc. At the moment, what
+   Buildroot does is clear and simple so its behaviour is very
+   reliable and it is easy to support users. If we start telling users
+   that the configuration changes done in menuconfig are applied after
Quoted 2 lines

"If we start telling users that" may be too conversational. Maybe "If we claim that"?

I've rephrased that to read:
If configuration changes done in menuconfig are applied after the next make [...]

Quoted 3 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 175

Quoted 3 lines
docs/manual/faq-troubleshooting.txt +3 -0
+   the next make, then it has to work correctly and properly in all
+   situations, and not have some bizarre corner cases. We fear bug
+   reports like "I have enabled package A, B and C, then ran make,

I've rephrased "We fear bug reports like [...]" to read:
The risk is to get bug reports like [...]

Quoted 8 lines
Quoted 8 lines
docs/manual/faq-troubleshooting.txt +8 -0
+   then disabled package C and enabled package D and ran make, then
+   re-enabled package C and enabled package E and then there is a
+   build failure". Or worse "I did some configuration, then built,
+   then did some changes, built, some more changes, built, some more
+   changes, built, and now it fails, but I don't remember all the
+   changes I did and in which order". This will be impossible to
+   support.
+
Quoted 8 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 189

Quoted 6 lines
docs/manual/faq-troubleshooting.txt +6 -0
+For all these reasons, the conclusion is that adding tracking of
+installed files to remove them when the package is unselected, or to
+generate a repository of binary packages, is something that is very
+hard to achieve reliably and will add a lot of complexity.
+
+On this matter, the Buildroot developpers make these position statements:

developpers -> developers

Fixed.

Quoted 3 lines
Quoted 1 line

+ * Buildroot strives at making it easy to generate a root filesystem

strives at making -> strives to make

Fixed.

Quoted 9 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 198

Quoted 7 lines
docs/manual/faq-troubleshooting.txt +7 -0
+   (hence the name, by the way). That is what we want to make Buildroot
+   good at: building root filesystems.
+
+ * Buildroot is not meant to be a distribution (or rather, a distribution
+   generator). It is the opinion of most Buildroot developers that this
+   is not a goal we should pursue. We believe that there are other tools
+   better suited to generate a distro than Buildroot is. For example,

than Buildroot is. -> than Buildroot.

In fact, not. This is a (double?) ellipsis of the adjectival clause 'suited to generate a distro', ellipsis which stands for (the parts between {} are the two ellipsises):
[...], than Buildroot is {suited to {generate a distro}}.

If there were no adjectival clause, we would write:
There are other tools better than Buildroot.

Thanks for the review!

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#4 lore raw

PATCH 1/1 manual/faq: add section about why no binary packages

Yann,

Quoted 19 lines · Yann E. MORIN

Danomi, All,

On 2014-02-17 15:21 -0500, Danomi Manchego spake thusly:

Quoted 1 line

Nit-picky writing editorial - no issue with the actual content ...

Thank you!

My answers below...

Quoted 1 line · Yann E. MORIN

[--SNIP--]

Quoted 7 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 119

Quoted 5 lines

+[[faq-no-binary-packages]]

docs/manual/faq-troubleshooting.txt +4 -0
+Why doesn't Buildroot generate binary packages (.deb, .ipkg...)?
+----------------------------------------------------------------
+
+One feature that is often discussed on the Buildroot list, is the

no comma

Quoted 10 lines

Fixed.

Quoted 8 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 125

Quoted 6 lines
docs/manual/faq-troubleshooting.txt +6 -0
+the general topic of "package management". To summarize, the idea
+would be to add some tracking of which Buildroot package installs
+what files, with the goals of:
+
+ * Being able to remove files installed by a package when this package
+   gets unselected from the menuconfig ;

; -> .

It seems strange to me that a bulleted list is also considered a complete sentence. I think of bulleted lists like an unsorted list of items, as opposed to a numbered list where numbers have meaning. But it's your party, keep as you like.

Quoted 31 lines

I am not so sure: this is a list, and to me the second bullet-point is included in that sentence.

Quoted 6 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 129

Quoted 3 lines

+ * Ultimately, be able to generate binary packages (ipk or other

docs/manual/faq-troubleshooting.txt +2 -0
+   format) that can be installed on the target without re-generating a
+   new root filesystem image.

There's some tense inconsistency here between the first point ("Being able to...") and the second ("be able to...")

Fixed.

Quoted 9 lines
Quoted 9 lines
docs/manual/faq-troubleshooting.txt +9 -0
+In general, most people think it is easy to do: just track which package
+installed what and remove it when the package is unselected. However, it
+is much more complicated than that:
+
+ * It is not only about the +target/+ directory, but also the sysroot in
+   +host/usr/<tuple>/sysroot+ and the +host/+ directory itself. All files
+   installed in those directories by various packages must be tracked.
+
+ * When a package is removed, it is not sufficient to remove just the
Quoted 8 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 145

Quoted 6 lines
docs/manual/faq-troubleshooting.txt +6 -0
+   files it installed. One must also remove all its reverse
+   dependencies (i.e packages relying on it) and rebuild all those
+   packages. For example, package A depends optionally on the OpenSSL
+   library. Both are selected, and Buildroot is built. Package A is
+   built with crypto support using OpenSSL. Later on, OpenSSL gets
+   unselected from the configuration, but package A remains (since

remains (since ,,, possible). -> remains. (Since ... possible.)

Quoted 2 lines

The phrase between parentheses is part of the previous sentence, so no upper-case is needed for 'since'.

I did not interpret the parenthetical content as part of the preceding sentence - rather it seems like a complete sentence on its own. Therefore, I thought that it should be apart from the first sentence, with the the period inside the parenthesis. If the parenthetic content is really a part of the preceding sentence and does not stand alone, then I would think that the period should be outside of the parenthesis - as you originally had it. But I won't press this point any further.

Danomi -

Quoted 113 lines

However, the english grammar rules dictate that the terminating punctuation be inside the parentheses, not outside. Fixed.

Quoted 8 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 151

Quoted 6 lines
docs/manual/faq-troubleshooting.txt +6 -0
+   OpenSSL is an optional dependency, this is possible). If you just
+   remove the OpenSSL files, then the files installed by package A are
+   broken: they use a library that is no longer present on the
+   target. Technically, it is possible to do this (the prototype that
+   Lionel Landwerlin and Thomas Petazzoni have worked on started to do
+   this), but it is difficult and adds a fair bit of complexity.

"a fair bit of" may be colloquial. Maybe change to "a lot of"?

Well, I like the turn of phrase, as it emphasises the fact that it is not easy (see eg. http://www.merriam-webster.com/dictionary/fair
defintion 10b: moderately numerous, large, or significant).

Quoted 12 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 163

Quoted 11 lines

+ * In addition to the previous problem, there is the case where the

docs/manual/faq-troubleshooting.txt +10 -0
+   optional dependency is not even known to Buildroot. For example,
+   package A in version 1.0 never used OpenSSL, but in version 2.0 it
+   automatically uses OpenSSL if available. If the Buildroot .mk file
+   hasn't been updated to take this into account, then package A will
+   not be part of the reverse dependencies of OpenSSL and will not be
+   removed and rebuilt when OpenSSL is removed. For sure, the .mk file
+   of package A should be fixed to mention this optional dependency,
+   but in the mean time, you can have non-reproducible behaviors.
+
+ * The whole idea is also to allow changes in the menuconfig to be
Quoted 1 line

Too conversational: "whole idea". Maybe "idea" or "implied goal"?

I've rephrased that to read:
The request is to also allow changes in the menuconfig [...]

Quoted 10 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 172

Quoted 9 lines
docs/manual/faq-troubleshooting.txt +9 -0
+   applied on the output directory without having to rebuild
+   everything from scratch. However, this is very difficult to achieve
+   in a reliable way: what happens when the suboptions of a package
+   are changed (we would have to detect this, and rebuild the package
+   from scratch and potentially all its reverse dependencies), what
+   happens if toolchain options are changed, etc. At the moment, what
+   Buildroot does is clear and simple so its behaviour is very
+   reliable and it is easy to support users. If we start telling users
+   that the configuration changes done in menuconfig are applied after
Quoted 2 lines

"If we start telling users that" may be too conversational. Maybe "If we claim that"?

I've rephrased that to read:
If configuration changes done in menuconfig are applied after the next make [...]

Quoted 3 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 175

Quoted 3 lines
docs/manual/faq-troubleshooting.txt +3 -0
+   the next make, then it has to work correctly and properly in all
+   situations, and not have some bizarre corner cases. We fear bug
+   reports like "I have enabled package A, B and C, then ran make,

I've rephrased "We fear bug reports like [...]" to read:
The risk is to get bug reports like [...]

Quoted 8 lines
Quoted 8 lines
docs/manual/faq-troubleshooting.txt +8 -0
+   then disabled package C and enabled package D and ran make, then
+   re-enabled package C and enabled package E and then there is a
+   build failure". Or worse "I did some configuration, then built,
+   then did some changes, built, some more changes, built, some more
+   changes, built, and now it fails, but I don't remember all the
+   changes I did and in which order". This will be impossible to
+   support.
+
Quoted 8 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 189

Quoted 6 lines
docs/manual/faq-troubleshooting.txt +6 -0
+For all these reasons, the conclusion is that adding tracking of
+installed files to remove them when the package is unselected, or to
+generate a repository of binary packages, is something that is very
+hard to achieve reliably and will add a lot of complexity.
+
+On this matter, the Buildroot developpers make these position statements:

developpers -> developers

Fixed.

Quoted 3 lines
Quoted 1 line

+ * Buildroot strives at making it easy to generate a root filesystem

strives at making -> strives to make

Fixed.

Quoted 9 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 198

Quoted 7 lines
docs/manual/faq-troubleshooting.txt +7 -0
+   (hence the name, by the way). That is what we want to make Buildroot
+   good at: building root filesystems.
+
+ * Buildroot is not meant to be a distribution (or rather, a distribution
+   generator). It is the opinion of most Buildroot developers that this
+   is not a goal we should pursue. We believe that there are other tools
+   better suited to generate a distro than Buildroot is. For example,

than Buildroot is. -> than Buildroot.

In fact, not. This is a (double?) ellipsis of the adjectival clause 'suited to generate a distro', ellipsis which stands for (the parts between {} are the two ellipsises):
[...], than Buildroot is {suited to {generate a distro}}.

If there were no adjectival clause, we would write:
There are other tools better than Buildroot.

Thanks for the review!

Regards,
Yann E. MORIN.

--
.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |

'------------------------------^-------^------------------^--------------------'

#5 lore raw

PATCH 1/1 manual/faq: add section about why no binary packages

Quoted 32 lines · Danomi Manchego

Yann,

Quoted 21 lines · Yann E. MORIN
Quoted 21 lines

Danomi, All,

On 2014-02-17 15:21 -0500, Danomi Manchego spake thusly:

Quoted 1 line
Quoted 1 line

Nit-picky writing editorial - no issue with the actual content ...

Thank you!

My answers below...

Quoted 1 line
Quoted 1 line · Yann E. MORIN

[--SNIP--]

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 119

Quoted 5 lines

+[[faq-no-binary-packages]]

docs/manual/faq-troubleshooting.txt +4 -0
+Why doesn't Buildroot generate binary packages (.deb, .ipkg...)?
+----------------------------------------------------------------
+
+One feature that is often discussed on the Buildroot list, is the

no comma

Fixed.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 125

Quoted 6 lines
docs/manual/faq-troubleshooting.txt +6 -0
+the general topic of "package management". To summarize, the idea
+would be to add some tracking of which Buildroot package installs
+what files, with the goals of:
+
+ * Being able to remove files installed by a package when this package
+   gets unselected from the menuconfig ;

; -> .

Quoted 4 lines

It seems strange to me that a bulleted list is also considered a complete sentence. I think of bulleted lists like an unsorted list of items, as opposed to a numbered list where numbers have meaning. But it's your party, keep as you like.

If you have a colon in front of it and a semicolon at the end, then the
sentence shouldn't be capitalized. So:

with the goals of:

* being able to remove files;
* being able to do the other thing.

Regards,
Arnout

[snip]

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
#6 lore raw

PATCH 1/1 manual/faq: add section about why no binary packages

In reply to Yann E. MORIN

Hi Yann,

Quoted 59 lines · Yann E. MORIN

From: "Yann E. MORIN" <[email protected]>

It comes up every now and then on the list, so better be prepared to point at the manual, rather than rehash the same every time.

Most of the chapter is a copy-paste of the report from the Buildroot Developpers Day in Pragues, 2011-10-28:

    http://lists.busybox.net/pipermail/buildroot/2011-November/047229.html

We consider the opinions expressed in that report to be still valid now, two years later.

Cc Samuel Martin <[email protected]>
Cc Peter Korsgaard <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Thomas Petazzoni <[email protected]>
Cc Thomas De Schampheleire <[email protected]>
Cc Baruch Siach <[email protected]>
---
 docs/manual/faq-troubleshooting.txt | 91 +++++++++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)
docs/manual/faq-troubleshooting.txt +28 -0
index 4e0612b..636b3b6 100644
@@ -111,3 +111,94 @@ directory as the new root, will most likely fail.
 If you want to run the target filesystem inside a chroot, or as an NFS
 root, then use the tarball image generated in +images/+ and extract it
 as root.
+
+[[faq-no-binary-packages]]
+Why doesn't Buildroot generate binary packages (.deb, .ipkg...)?
+----------------------------------------------------------------
+
+One feature that is often discussed on the Buildroot list, is the
+the general topic of "package management". To summarize, the idea
+would be to add some tracking of which Buildroot package installs
+what files, with the goals of:
+
+ * Being able to remove files installed by a package when this package
+   gets unselected from the menuconfig ;
+
+ * Ultimately, be able to generate binary packages (ipk or other
+   format) that can be installed on the target without re-generating a
+   new root filesystem image.
+
+In general, most people think it is easy to do: just track which package
+installed what and remove it when the package is unselected. However, it
+is much more complicated than that:
+
+ * It is not only about the +target/+ directory, but also the sysroot in
+   +host/usr/<tuple>/sysroot+ and the +host/+ directory itself. All files
+   installed in those directories by various packages must be tracked.
+
+ * When a package is removed, it is not sufficient to remove just the
+   files it installed. One must also remove all its reverse
+   dependencies (i.e packages relying on it) and rebuild all those

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 151

Quoted 10 lines
docs/manual/faq-troubleshooting.txt +10 -0
+   packages. For example, package A depends optionally on the OpenSSL
+   library. Both are selected, and Buildroot is built. Package A is
+   built with crypto support using OpenSSL. Later on, OpenSSL gets
+   unselected from the configuration, but package A remains (since
+   OpenSSL is an optional dependency, this is possible). If you just
+   remove the OpenSSL files, then the files installed by package A are
+   broken: they use a library that is no longer present on the
+   target. Technically, it is possible to do this (the prototype that
+   Lionel Landwerlin and Thomas Petazzoni have worked on started to do
+   this), but it is difficult and adds a fair bit of complexity.

I'm a bit skeptical about mentioning the existence of such a prototype. People may start asking for these patches, which may obviously not apply anymore.
I would just say that, though this is technically doable, it adds a lot of complexity to Buildroot, which goes against the simplicity we tend to keep.

Quoted 8 lines
docs/manual/faq-troubleshooting.txt +8 -0
+
+ * In addition to the previous problem, there is the case where the
+   optional dependency is not even known to Buildroot. For example,
+   package A in version 1.0 never used OpenSSL, but in version 2.0 it
+   automatically uses OpenSSL if available. If the Buildroot .mk file
+   hasn't been updated to take this into account, then package A will
+   not be part of the reverse dependencies of OpenSSL and will not be
+   removed and rebuilt when OpenSSL is removed. For sure, the .mk file

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 169

Quoted 10 lines
docs/manual/faq-troubleshooting.txt +10 -0
+   of package A should be fixed to mention this optional dependency,
+   but in the mean time, you can have non-reproducible behaviors.
+
+ * The whole idea is also to allow changes in the menuconfig to be
+   applied on the output directory without having to rebuild
+   everything from scratch. However, this is very difficult to achieve
+   in a reliable way: what happens when the suboptions of a package
+   are changed (we would have to detect this, and rebuild the package
+   from scratch and potentially all its reverse dependencies), what
+   happens if toolchain options are changed, etc. At the moment, what

What should be done when the toolchain options are changed is already known and documented ;) :
http://nightly.buildroot.org/manual.html#full-rebuild

Quoted 36 lines
docs/manual/faq-troubleshooting.txt +35 -1
+   Buildroot does is clear and simple so its behaviour is very
+   reliable and it is easy to support users. If we start telling users
+   that the configuration changes done in menuconfig are applied after
+   the next make, then it has to work correctly and properly in all
+   situations, and not have some bizarre corner cases. We fear bug
+   reports like "I have enabled package A, B and C, then ran make,
+   then disabled package C and enabled package D and ran make, then
+   re-enabled package C and enabled package E and then there is a
+   build failure". Or worse "I did some configuration, then built,
+   then did some changes, built, some more changes, built, some more
+   changes, built, and now it fails, but I don't remember all the
+   changes I did and in which order". This will be impossible to
+   support.
+
+For all these reasons, the conclusion is that adding tracking of
+installed files to remove them when the package is unselected, or to
+generate a repository of binary packages, is something that is very
+hard to achieve reliably and will add a lot of complexity.
+
+On this matter, the Buildroot developpers make these position statements:
+
+ * Buildroot strives at making it easy to generate a root filesystem
+   (hence the name, by the way). That is what we want to make Buildroot
+   good at: building root filesystems.
+
+ * Buildroot is not meant to be a distribution (or rather, a distribution
+   generator). It is the opinion of most Buildroot developers that this
+   is not a goal we should pursue. We believe that there are other tools
+   better suited to generate a distro than Buildroot is. For example,
+   http://openembedded.org/[Open Embedded], or https://openwrt.org/[openWRT],
+   are such tools.
+
+ * We prefer to push Buildroot in a direction that makes it easy (or even
+   easier) to generate complete root filesystems. This is what makes
+   Buildroot stands out in the crowd (among other things, of course!).
--
Quoted 2 lines

1.8.1.2

Regards,

Signature · 1 line
Samuel
#7 lore raw

PATCH 1/1 manual/faq: add section about why no binary packages

In reply to Yann E. MORIN

Dear Yann E. MORIN,

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 135

Quoted 5 lines · Yann E. MORIN
docs/manual/faq-troubleshooting.txt +5 -0
+In general, most people think it is easy to do: just track which package
+installed what and remove it when the package is unselected. However, it
+is much more complicated than that:
+
+ * It is not only about the +target/+ directory, but also the sysroot in

"It is not only" is a bit vague. Proposal:

* The problem is not limited to files installed in the +target/+
directory: the toolchain sysroot in +host/usr/<tuple>/sysroot+ as well as host tools installed in +host/+ should also be tracked.

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail.

Quoted 5 lines
docs/manual/faq-troubleshooting.txt +5 -0
+   +host/usr/<tuple>/sysroot+ and the +host/+ directory itself. All files
+   installed in those directories by various packages must be tracked.
+
+ * When a package is removed, it is not sufficient to remove just the
+   files it installed.

When a package is unselected from the configuration, it is not sufficient to...

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 151

Quoted 12 lines

One must also remove all its reverse

docs/manual/faq-troubleshooting.txt +11 -0
+   dependencies (i.e packages relying on it) and rebuild all those
+   packages. For example, package A depends optionally on the OpenSSL
+   library. Both are selected, and Buildroot is built. Package A is
+   built with crypto support using OpenSSL. Later on, OpenSSL gets
+   unselected from the configuration, but package A remains (since
+   OpenSSL is an optional dependency, this is possible). If you just
+   remove the OpenSSL files, then the files installed by package A are
+   broken: they use a library that is no longer present on the
+   target. Technically, it is possible to do this (the prototype that
+   Lionel Landwerlin and Thomas Petazzoni have worked on started to do
+   this), but it is difficult and adds a fair bit of complexity.

As Samuel suggested, I don't think mentioning a prototype or even the names of people who did the prototype belong to an official documentation.

Quoted 42 lines

+ * In addition to the previous problem, there is the case where the

docs/manual/faq-troubleshooting.txt +41 -0
+   optional dependency is not even known to Buildroot. For example,
+   package A in version 1.0 never used OpenSSL, but in version 2.0 it
+   automatically uses OpenSSL if available. If the Buildroot .mk file
+   hasn't been updated to take this into account, then package A will
+   not be part of the reverse dependencies of OpenSSL and will not be
+   removed and rebuilt when OpenSSL is removed. For sure, the .mk file
+   of package A should be fixed to mention this optional dependency,
+   but in the mean time, you can have non-reproducible behaviors.
+
+ * The whole idea is also to allow changes in the menuconfig to be
+   applied on the output directory without having to rebuild
+   everything from scratch. However, this is very difficult to achieve
+   in a reliable way: what happens when the suboptions of a package
+   are changed (we would have to detect this, and rebuild the package
+   from scratch and potentially all its reverse dependencies), what
+   happens if toolchain options are changed, etc. At the moment, what
+   Buildroot does is clear and simple so its behaviour is very
+   reliable and it is easy to support users. If we start telling users
+   that the configuration changes done in menuconfig are applied after
+   the next make, then it has to work correctly and properly in all
+   situations, and not have some bizarre corner cases. We fear bug
+   reports like "I have enabled package A, B and C, then ran make,
+   then disabled package C and enabled package D and ran make, then
+   re-enabled package C and enabled package E and then there is a
+   build failure". Or worse "I did some configuration, then built,
+   then did some changes, built, some more changes, built, some more
+   changes, built, and now it fails, but I don't remember all the
+   changes I did and in which order". This will be impossible to
+   support.
+
+For all these reasons, the conclusion is that adding tracking of
+installed files to remove them when the package is unselected, or to
+generate a repository of binary packages, is something that is very
+hard to achieve reliably and will add a lot of complexity.
+
+On this matter, the Buildroot developpers make these position statements:
+
+ * Buildroot strives at making it easy to generate a root filesystem
+   (hence the name, by the way). That is what we want to make Buildroot
+   good at: building root filesystems.
+

docs/manual/faq-troubleshooting.txt directory as the new root, will most likely fail. · line 204

Quoted 10 lines
docs/manual/faq-troubleshooting.txt +10 -0
+ * Buildroot is not meant to be a distribution (or rather, a distribution
+   generator). It is the opinion of most Buildroot developers that this
+   is not a goal we should pursue. We believe that there are other tools
+   better suited to generate a distro than Buildroot is. For example,
+   http://openembedded.org/[Open Embedded], or https://openwrt.org/[openWRT],
+   are such tools.
+
+ * We prefer to push Buildroot in a direction that makes it easy (or even
+   easier) to generate complete root filesystems. This is what makes
+   Buildroot stands out in the crowd (among other things, of course!).

Additional item (probably the most important one) :

* We believe that for most embedded Linux systems, binary packages are not necessary, and potentially harmful. When binary packages are used, it means that the system can be partially upgraded, which creates an enormous number of possible combinations of package versions that should be tested before doing the upgrade on the embedded device. On the other hand, by doing complete system upgrades by upgrading the entire root filesystem image at once, you guarantee that the image you deploy to you embedded system is really the one you have tested and validated.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com