kconfig: Kconfiglib: a flexible Python Kconfig parser

Hi,

This is the initial release of Kconfiglib: a Python library for scripting, debugging, and extracting information from Kconfig-based configuration systems.  It can be used to programmatically generate a .config when the '*conf' tools are too inflexible, to quickly find out interesting information about a Kconfig configuration such as dependency relations between symbols and where undefined symbols are referenced, and in applications that need to parse and extract information from Kconfig files.

For a much longer introduction including multiple examples, see arch/kconfig/kconfiglib.py.

Have fun!

Signed-off-by Ulf Magnusson <[email protected]>

---
Convenience links:

Documentation, generated from kconfiglib.py with pydoc -w:
http://dl.dropbox.com/u/10406197/kconfiglib.html

Examples as separate files:
http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz

The patch should be preferably be applied to a recent kernel, i.e. Linus's (2.6.38-rc3 at the time of writing).  Due to recent Kconfig changes, the kconfigtest.py test suite - which compares output character-for-character - will indicate failure on older (a few months old) kernels versions even though the outputs are functionally equivalent.

 Documentation/kbuild/kconfig-language.txt |    5 + Documentation/kbuild/kconfig.txt          |    8 +
 README                                    |   13 + scripts/kconfig/Makefile                  |   26 +- scripts/kconfig/kconfiglib.py             | 3918 +++++++++++++++++++++++++++++ scripts/kconfig/kconfigtest.py            |  396 +++

 6 files changed, 4365 insertions(+), 1 deletions(-)

 create mode 100644 scripts/kconfig/kconfiglib.py create mode 100644 scripts/kconfig/kconfigtest.py [...]

Hi,

Quoted 11 lines · Ulf Magnusson

Hi,

This is the initial release of Kconfiglib: a Python library for scripting, debugging, and extracting information from Kconfig-based configuration systems.  It can be used to programmatically generate a .config when the '*conf' tools are too inflexible, to quickly find out interesting information about a Kconfig configuration such as dependency relations between symbols and where undefined symbols are referenced, and in applications that need to parse and extract information from Kconfig files.

Does that mean that when a kconfig-language change will happen, one will not only must have to be Lex/YaCC and C fluent, but also, perl and python ? That seem to be a lot of duplication to me.

I'd rather see the backend be changed so that it could be used with SWIG to generate language bindings. I had WIP in this area, but changes needed are rather intrusive. That said, the testsuite might be interesting, I've been thinking about that for quite some time.

- Arnaud

Quoted 32 lines

For a much longer introduction including multiple examples, see arch/kconfig/kconfiglib.py.

Have fun!

Signed-off-by Ulf Magnusson <[email protected]>

---
Convenience links:

Documentation, generated from kconfiglib.py with pydoc -w:
http://dl.dropbox.com/u/10406197/kconfiglib.html

Examples as separate files:
http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz

The patch should be preferably be applied to a recent kernel, i.e. Linus's (2.6.38-rc3 at the time of writing).  Due to recent Kconfig changes, the kconfigtest.py test suite - which compares output character-for-character - will indicate failure on older (a few months old) kernels versions even though the outputs are functionally equivalent.

 Documentation/kbuild/kconfig-language.txt |    5 + Documentation/kbuild/kconfig.txt          |    8 +
 README                                    |   13 + scripts/kconfig/Makefile                  |   26 +- scripts/kconfig/kconfiglib.py             | 3918 +++++++++++++++++++++++++++++ scripts/kconfig/kconfigtest.py            |  396 +++

 6 files changed, 4365 insertions(+), 1 deletions(-)

 create mode 100644 scripts/kconfig/kconfiglib.py create mode 100644 scripts/kconfig/kconfigtest.py [...]

Hi,

This is the initial release of Kconfiglib: a Python library for scripting, debugging, and extracting information from Kconfig-based configuration systems. It can be used to programmatically generate a .config when the '*conf' tools are too inflexible, to quickly find out interesting information about a Kconfig configuration such as dependency relations between symbols and where undefined symbols are referenced, and in applications that need to parse and extract information from Kconfig files.

For a much longer introduction including multiple examples, see arch/kconfig/kconfiglib.py.

Have fun!

Signed-off-by Ulf Magnusson <[email protected]>

---
Convenience links:

Documentation, generated from kconfiglib.py with pydoc -w:
http://dl.dropbox.com/u/10406197/kconfiglib.html

Examples as separate files:
http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz

The patch should be preferably be applied to a recent kernel, i.e. Linus's (2.6.38-rc3 at the time of writing). Due to recent Kconfig changes, the kconfigtest.py test suite - which compares output character-for-character - will indicate failure on older (a few months old) kernels versions even though the outputs are functionally equivalent.

[...]

Quoted 33 lines · Ulf Magnusson

Hi,

This is the initial release of Kconfiglib: a Python library for scripting, debugging, and extracting information from Kconfig-based configuration systems. It can be used to programmatically generate a .config when the '*conf' tools are too inflexible, to quickly find out interesting information about a Kconfig configuration such as dependency relations between symbols and where undefined symbols are referenced, and in applications that need to parse and extract information from Kconfig files.

For a much longer introduction including multiple examples, see arch/kconfig/kconfiglib.py.

Have fun!

Signed-off-by Ulf Magnusson <[email protected]>

---
Convenience links:

Documentation, generated from kconfiglib.py with pydoc -w:
http://dl.dropbox.com/u/10406197/kconfiglib.html

Examples as separate files:
http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz

The patch should be preferably be applied to a recent kernel, i.e. Linus's (2.6.38-rc3 at the time of writing). Due to recent Kconfig changes, the kconfigtest.py test suite - which compares output character-for-character - will indicate failure on older (a few months old) kernels versions even though the outputs are functionally equivalent.

Quoted 1 line

[...]

Seems linux-kbuild and linux-doc won't accept the patch (too large?), so here's a link to the message on linux-kernel:
https://lkml.org/lkml/2011/2/1/439

/Ulf Magnusson

Hi,

This is the initial release of Kconfiglib: a Python library for scripting, debugging, and extracting information from Kconfig-based configuration systems. It can be used to programmatically generate a .config when the '*conf' tools are too inflexible, to quickly find out interesting information about a Kconfig configuration such as dependency relations between symbols and where undefined symbols are referenced, and in applications that need to parse and extract information from Kconfig files.

For a much longer introduction including multiple examples, see arch/kconfig/kconfiglib.py.

Have fun!

Signed-off-by Ulf Magnusson <[email protected]>

---
Convenience links:

Documentation, generated from kconfiglib.py with pydoc -w:
http://dl.dropbox.com/u/10406197/kconfiglib.html

Examples as separate files:
http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz

The patch should be preferably be applied to a recent kernel, i.e. Linus's (2.6.38-rc3 at the time of writing). Due to recent Kconfig changes, the kconfigtest.py test suite - which compares output character-for-character - will indicate failure on older (a few months old) kernels versions even though the outputs are functionally equivalent.

 Documentation/kbuild/kconfig-language.txt |    5 +
 Documentation/kbuild/kconfig.txt          |    8 +
 README                                    |   13 +
 scripts/kconfig/Makefile                  |   26 +-
 scripts/kconfig/kconfiglib.py             | 3918 +++++++++++++++++++++++++++++
 scripts/kconfig/kconfigtest.py            |  396 +++
 6 files changed, 4365 insertions(+), 1 deletions(-)
 create mode 100644 scripts/kconfig/kconfiglib.py
 create mode 100644 scripts/kconfig/kconfigtest.py

README +6 -0

index 1b81d28..bb5e68f 100644
@@ -196,6 +196,19 @@ CONFIGURING the kernel:
 			   values to 'n' as much as possible.
 	"make randconfig"  Create a ./.config file by setting symbol
 			   values to random values.
+	"make scriptconfig SCRIPT=<path to script>" Run a Kconfiglib
+			   script (see scripts/kconfig/kconfiglib.py).  This
+			   can be used to programatically generate a
+			   ./.config, and for applications that need to
+			   extract information from Kconfig files.
+	"make iscriptconfig" Launch an interactive Python shell

Quoted 6 lines

The patch should be preferably be applied to a recent kernel, i.e. Linus's (2.6.38-rc3 at the time of writing). Due to recent Kconfig changes, the kconfigtest.py test suite - which compares output character-for-character - will indicate failure on older (a few months old) kernels versions even though the outputs are functionally equivalent.

Quoted 9 lines

 Documentation/kbuild/kconfig-language.txt |    5 +
 Documentation/kbuild/kconfig.txt          |    8 +
 README                                    |   13 +
 scripts/kconfig/Makefile                  |   26 +-
 scripts/kconfig/kconfiglib.py             | 3918 +++++++++++++++++++++++++++++
 scripts/kconfig/kconfigtest.py            |  396 +++
 6 files changed, 4365 insertions(+), 1 deletions(-)
 create mode 100644 scripts/kconfig/kconfiglib.py
 create mode 100644 scripts/kconfig/kconfigtest.py

Hello Ulf!

I'm working on pretty same thing also in python... Now I'm sad, because you made it and I don't, and also my code looks like dung in comparison to yours. How long did you make this?

BTW. "SLOW_WORK" symbol used in drivers/gpu/drm/Kconfig in "menucofnig DRM" as one of selects isn't an unnecessary symbol? I can't find any SLOW_WORK entry.

Quoted 1 line

Quoted 1 line

c["SLOW_WORK"].is_defined()

Quoted 6 lines · Randy Dunlap

Quoted 6 lines

The patch should be preferably be applied to a recent kernel, i.e. Linus's (2.6.38-rc3 at the time of writing). Due to recent Kconfig changes, the kconfigtest.py test suite - which compares output character-for-character - will indicate failure on older (a few months old) kernels versions even though the outputs are functionally equivalent.

Quoted 9 lines

Quoted 9 lines

 Documentation/kbuild/kconfig-language.txt |    5 +
 Documentation/kbuild/kconfig.txt          |    8 +
 README                                    |   13 +
 scripts/kconfig/Makefile                  |   26 +-
 scripts/kconfig/kconfiglib.py             | 3918 +++++++++++++++++++++++++++++
 scripts/kconfig/kconfigtest.py            |  396 +++
 6 files changed, 4365 insertions(+), 1 deletions(-)
 create mode 100644 scripts/kconfig/kconfiglib.py
 create mode 100644 scripts/kconfig/kconfigtest.py

Wait, I thought this was an extra standalone library. Are you saying you want to make it so Linux will no longer compile on a build machine that doesn't have Python installed?

If this is merely an extra developer tool ala bloat-o-meter and checkpatch.pl then it's merely uninteresting to me. (I myself wrote a quick and dirty http://kernel.org/doc/make/menuconfig2html.py to generate http://kernel.org/doc/menuconfig/x86.html and friends years ago, and I still run it to update that once in a while. There's not much to it.)

But adding new prerequities to a build machine would be really annoying for my use cases.

Rob

Quoted 58 lines · On Wed, 2 Feb 2011 00:27:55 +0100 Ulf Magnusson wrote:

Hi,

This is the initial release of Kconfiglib: a Python library for scripting, debugging, and extracting information from Kconfig-based configuration systems. It can be used to programmatically generate a .config when the '*conf' tools are too inflexible, to quickly find out interesting information about a Kconfig configuration such as dependency relations between symbols and where undefined symbols are referenced, and in applications that need to parse and extract information from Kconfig files.

For a much longer introduction including multiple examples, see arch/kconfig/kconfiglib.py.

Have fun!

Signed-off-by Ulf Magnusson <[email protected]>

---
Convenience links:

Documentation, generated from kconfiglib.py with pydoc -w:
http://dl.dropbox.com/u/10406197/kconfiglib.html

Examples as separate files:
http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz

The patch should be preferably be applied to a recent kernel, i.e. Linus's (2.6.38-rc3 at the time of writing). Due to recent Kconfig changes, the kconfigtest.py test suite - which compares output character-for-character - will indicate failure on older (a few months old) kernels versions even though the outputs are functionally equivalent.

 Documentation/kbuild/kconfig-language.txt |    5 +
 Documentation/kbuild/kconfig.txt          |    8 +
 README                                    |   13 +
 scripts/kconfig/Makefile                  |   26 +-
 scripts/kconfig/kconfiglib.py             | 3918 +++++++++++++++++++++++++++++
 scripts/kconfig/kconfigtest.py            |  396 +++
 6 files changed, 4365 insertions(+), 1 deletions(-)
 create mode 100644 scripts/kconfig/kconfiglib.py
 create mode 100644 scripts/kconfig/kconfigtest.py

README +6 -0

index 1b81d28..bb5e68f 100644
@@ -196,6 +196,19 @@ CONFIGURING the kernel:
 			   values to 'n' as much as possible.
 	"make randconfig"  Create a ./.config file by setting symbol
 			   values to random values.
+	"make scriptconfig SCRIPT=<path to script>" Run a Kconfiglib
+			   script (see scripts/kconfig/kconfiglib.py).  This
+			   can be used to programatically generate a
+			   ./.config, and for applications that need to
+			   extract information from Kconfig files.
+	"make iscriptconfig" Launch an interactive Python shell

README CONFIGURING the kernel: · line 214

Quoted 10 lines

README +7 -0

+			   for running Kconfiglib on the architecture's
+			   Kconfig configuration.  The kconfiglib and sys
+			   (for sys.argv[1] - the base Kconfig file) modules
+			   will be imported automatically, and a Config
+			   instance 'c' will be created for the architecture
+			   (using c = kconfiglib.Config(sys.argv[1])).
+
 
    You can find more information on using the Linux kernel config tools
    in Documentation/kbuild/kconfig.txt.

Hi Ulf,

This is interesting. I just wish I could read it. ;) I'll get over it.

1. It would be really Good to have "make scriptconfig SCRIPT=<path to script>" and "make iscriptconfig" (similar to above, but shortened) in "make help" output.

2. My first test (using your ex1.py script) failed because I used O=xx64 (build directory):

rddunlap@chimera:lnx-2638-rc3> make O=xx64 scriptconfig SCRIPT=~/pkg/kconfiglib/ex1.py
GEN /lnx/src/lnx-2638-rc3/xx64/Makefile
Traceback (most recent call last):
File "/home/rddunlap/pkg/kconfiglib/ex1.py", line 1, in <module>
import kconfiglib
ImportError: No module named kconfiglib
make[2]: *** [scriptconfig] Error 1
make[1]: *** [scriptconfig] Error 2
make: *** [sub-make] Error 2

Does kconfiglib support O=builddir generally? I can't tell that it does (yet) since all ex[1-7].py fail in this manner.
It needs to support/allow O=builddir.

Hi,

This is the initial release of Kconfiglib: a Python library for scripting, debugging, and extracting information from Kconfig-based configuration systems. It can be used to programmatically generate a .config when the '*conf' tools are too inflexible, to quickly find out interesting information about a Kconfig configuration such as dependency relations between symbols and where undefined symbols are referenced, and in applications that need to parse and extract information from Kconfig files.

For a much longer introduction including multiple examples, see arch/kconfig/kconfiglib.py.

Quoted 11 lines · Ulf Magnusson

Hi,

This is the initial release of Kconfiglib: a Python library for scripting, debugging, and extracting information from Kconfig-based configuration systems. It can be used to programmatically generate a .config when the '*conf' tools are too inflexible, to quickly find out interesting information about a Kconfig configuration such as dependency relations between symbols and where undefined symbols are referenced, and in applications that need to parse and extract information from Kconfig files.

Quoted 2 lines

For a much longer introduction including multiple examples, see arch/kconfig/kconfiglib.py.

Hi,

this looks like a very powerful tool, but I have a similar concern like Arnaud had - being completely standalone, it reimplements most of the C kconfig code. One option to reduce this duplication would be a swig wrapper, another one would be to let the C code parse the Kconfig files and write the required information in some digested form, that would be easier to parse by scripts. Something like:

$ scripts/kconfig/conf_inspect --kconfig=Kconfig --eval='FOO || BAR'
y
$ scripts/kconfig/conf_inspect ... --dump-symbols
config FOO
type: bool
valule: m
visible: y
prompt: "zzz"
depends: X & Y
select: Z
...
$ scripts/kconfig/conf_inspect ... --dump-symbols \
    --fields='depends,select,value'
config FOO
depends: X & Y
select: Z
value: m

config BAR
...

etc. The idea is that for instance instead of parsing the Kconfig files, the Python code could fill it's data structures by reading the flat dump provided by the C kconfig. There would be still lot to do in Python, e.g. parsing and evaluating expressions, but it would be a small step forward already. And people wanting to write quick&dirty scripts in bash/awk/perl would make use of the C code as well.

Quoted 44 lines · Michal Marek

Quoted 11 lines · Ulf Magnusson

Hi,

This is the initial release of Kconfiglib: a Python library for scripting, debugging, and extracting information from Kconfig-based configuration systems. It can be used to programmatically generate a .config when the '*conf' tools are too inflexible, to quickly find out interesting information about a Kconfig configuration such as dependency relations between symbols and where undefined symbols are referenced, and in applications that need to parse and extract information from Kconfig files.

Quoted 2 lines

For a much longer introduction including multiple examples, see arch/kconfig/kconfiglib.py.

Hi,

this looks like a very powerful tool, but I have a similar concern like Arnaud had - being completely standalone, it reimplements most of the C kconfig code. One option to reduce this duplication would be a swig wrapper, another one would be to let the C code parse the Kconfig files and write the required information in some digested form, that would be easier to parse by scripts. Something like:

$ scripts/kconfig/conf_inspect --kconfig=Kconfig --eval='FOO || BAR'
y
$ scripts/kconfig/conf_inspect ... --dump-symbols
config FOO
type: bool
valule: m
visible: y
prompt: "zzz"
depends: X & Y
select: Z
...
$ scripts/kconfig/conf_inspect ... --dump-symbols \
    --fields='depends,select,value'
config FOO
depends: X & Y
select: Z
value: m

config BAR
...

Quoted 7 lines

etc. The idea is that for instance instead of parsing the Kconfig files, the Python code could fill it's data structures by reading the flat dump provided by the C kconfig. There would be still lot to do in Python, e.g. parsing and evaluating expressions, but it would be a small step forward already. And people wanting to write quick&dirty scripts in bash/awk/perl would make use of the C code as well.

Offloading some of the parsing to a new tool would probably be doable, though some Kconfig subleties might complicate things. For example, symbols that appear within choices aren't seen as real selectable choice items if they depend on previous symbols in specific ways (see _determine_actual_items() in kconfiglib.py), and capturing this information in the output while at the same preserving the ordering of the symbols could be messy. I guess you could sort this out within Kconfiglib instead, though other tools might then have to reimplement the same (messy) rules.

I'm not so sure about offloading anything else this way. For dynamically updating symbol values in a script you would need to somehow preserve state between invocations (or pass huge lists of assignments back and forth), which could lead to an unwieldy protocol. You might need to go the SWIG route instead.