diet-kconfig: a script to trim unneeded kconfigs
Hi,
a topic that just came up in kernel summit is a script to remove unneeded kernel configs automatically to reduce the compile time. Incidentally, I already wrote such a script during the last SUSE hack week a couple of weeks ago, so I'd like to share here, hopefully to give an idea for further improvements.
The script checks the currently loaded modules and trims other CONFIG_XXX=m simply, and tries make oldconfig, and writes out the resultant .config in the current directory after some checks. You can specify the config file via option, as default, it reads from /proc/config.gz.
The script is VERY hackish. I should have begun with perl or whatever better script language, but I chose bash and co. So, don't expect much code quality. I'm no script guy after all :)
Takashi
---
#!/bin/sh
# this magic makes grep and co a lot faster export LANG=C
kroot=
subarch=$(uname -i)
oldconfig=/proc/config.gz
localversion=
modulelist=
optimize_smp=
optimize_memory=
outputfile=
usage () {
cat <<EOF
diet-kconfig -- Trim down kernel configs of unused modules
Usage: diet-kconfig [-options]
Create a kernel config file based on the running system
option:
-r ROOT Specifies the linux-kernel root, default: current dir -a ARCH Architecture (e.g. i386), default: running arch -c FILE Old config file, default: /proc/config.gz) -o FILE Output config file, default: kernel/.config -l LIST Module list, default: read lsmod -v VER Specifies the local version string, default: as is the original -s Optimize SMP -m Optimize memory model (for i386) -h This help
EOF
exit 1
}
while getopts r:a:c:o:l:v:pmh opt; do
case "$opt" in
r)
kroot="$OPTARG"
;;
a)
subarch="$OPTARG"
;;
c)
oldconfig="$OPTARG"
if [ ! -f "$oldconfig" ]; then
echo "Cannot find old config $oldconfig"
exit 1
fi
;;
o)
outputfile="$OPTARG"
;;
l)
modulelist="$OPTARG"
if [ ! -f "$modulelist" ]; then
echo "Cannot find module list file modulelist"
exit 1
fi
;;
v)
localversion="-$OPTARG"
;;
s)
optimize_smp=y
;;
m)
optimize_memory=y
;;
h)
usage
;;
*)
echo "Invalid option $opt"
usage
;;
esacdone
if [ -z "$kroot" ]; then
kdir=.
else
kdir="$kroot"fi
if [ ! -d "$kdir" -o ! -f "$kdir"/Makefile -o ! -f "$kdir"/MAINTAINERS ]; then
echo "Invalid kernel root directory $kdir"
exit 1
fi
case $subarch in
i?86|x86_64) arch=x86;;
*) arch=$subarch;;esac
tmpd=$(mktemp -q -d /tmp/dietkconf.XXXXXX)
cleanup () {
rm -rf $tmpd
exit 1
}
trap cleanup 0
curdir=$(pwd)
test -n "$kroot" && cd $kroot
# create a list of Makefile's find arch/$arch block crypto drivers fs ipc kernel lib mm net security sound virt \ -name 'Makefile' | xargs cat >> $tmpd/makefiles
# find configs to create the given module
find_kconfig () {
module=$1
modregex=$(echo $1 | sed -e's/_/[_-]/g')grep -E '^[[:space:]]*obj-\$\([A-Za-z0-9_]*\)[[:space:]]*[+:]=.*[[:space:]]'"$modregex"'\.o' $tmpd/makefiles | \ sed -e's/^.*://g' -e's/^[[:space:]]*obj-\$(\(.*\)).*$/\1/g' }
# set config no
kconfig_no () {
sed -i -e's/^\('$1'\)=.*/# \1 is not set/' $tmpd/kconfig
}# set config yes
kconfig_yes () {
sed -i -e's/# '$1' .*/'$1'=y/' $tmpd/kconfig
}# only previously selected modules with XXX marks
kconfig_mod () {
sed -i -e's/# '$1' XXX/'$1'=m/' $tmpd/kconfig
}get_old_config () {
case "$oldconfig" in
*.gz)
zcat $oldconfig
;;
*)
cat $oldconfig
esac
}list_modules () {
if [ -n "$modulelist" ]; then
awk '{if ($1 != "Module") {print $1}}' < $modulelist
else
lsmod | awk '{if ($1 != "Module") {print $1}}'
fi
}# copy the original kconfigs; reset all modules, but remember they were # enabled by marking XXX get_old_config | sed -e's/^\(CONFIG.*\)=m/# \1 XXX/g' > $tmpd/kconfig
# optimize some stuff (if given)
if [ "$optimize_smp" = "y" ]; then
cpus=$(cat /proc/cpuinfo | grep -q -c 'processor[[:space:]]*:')
if [ "$cpus" = 1 ]; then
kconfig_no CONFIG_SMP
else
kconfig_yes CONFIG_SMP
fi
fi
if [ "$subarch" = "i386" ]; then
if [ "$optimize_memory" = "y" ]; then
mem=$(free | grep 'Mem:' | awk '{print $2}')
if [ $mem -gt 4194304 ]; then
kconfig_yes CONFIG_HIGHMEM64G
kconfig_no CONFIG_HIGHMEM4G
kconfig_no CONFIG_NOHIGHMEM
kconfig_yes CONFIG_X86_PAE
elif [ $mem -gt 1048576 ]; then
kconfig_no CONFIG_HIGHMEM64G
kconfig_yes CONFIG_HIGHMEM4G
kconfig_no CONFIG_NOHIGHMEM
kconfig_no CONFIG_X86_PAE
else
kconfig_no CONFIG_HIGHMEM64G
kconfig_no CONFIG_HIGHMEM4G
kconfig_yes CONFIG_NOHIGHMEM
kconfig_no CONFIG_X86_PAE
fi
fi
fi# rewrite the local version
if [ -n "$localversion" ]; then
sed -i -e's/^CONFIG_LOCALVERSION=.*/CONFIG_LOCALVERSION="'$localversion'"/' $tmpd/kconfig
fi# find kconfig for each module
rm -f $noconfigs $okconfigs
list_modules | while read m; do
echo -n "Checking module $m..."
cfgs=$(find_kconfig $m)
if [ -z "$cfgs" ]; then
echo -n "N/A"
echo $m >> $tmpd/conf-no
else
echo -n "$m" >> $tmpd/conf-ok
for c in $cfgs; do
echo -n " $c"
echo -n " $c" >> $tmpd/conf-ok
kconfig_mod $c
done
echo >> $tmpd/conf-ok
fi
echo
done# if the previously =m doesn't enable any modules, it's a virtual one, which
# we should keep =m
for c in $(grep '^#.*XXX$' $tmpd/kconfig | sed -e's/^#.*\(CONFIG.*\) XXX/\1/g'); do
grep -q 'obj-\$('$c').*=.*\.o' $tmpd/makefiles || kconfig_mod $c
done# replace XXX to standard sed -i -e's/^\(#.*\) XXX/\1 is not set/g' $tmpd/kconfig
# reconfigure echo "Reconfiguring via make silentconfig" cd $curdir cp $tmpd/kconfig .config
if [ -n "$kroot" ]; then
make_args="-C $kroot O=$curdir"
else
make_args=
fi
make $make_args silentoldconfig ARCH=$arch
if [ -n "$outputfile" ]; then
echo "Copying the result to $outputfile"
cp .config $outputfile
fi
echo
echo "*** FINISHED ***"
echo
if [ -s $tmpd/conf-no ]; then
echo "Modules not configured:"
cat $tmpd/conf-no
fi
if [ -s $tmpd/conf-ok ]; then
while read m cfgs; do
found=
for c in $cfgs; do
if grep -q $c=m .config; then
found=y
break
fi
done
if [ -z "$found" ]; then
echo "Not enabled: module $m with config $cfgs"
fi
done < $tmpd/conf-ok
fi
exit 0
Quoted 8 lines · Takashi Iwai
Hi,
a topic that just came up in kernel summit is a script to remove unneeded kernel configs automatically to reduce the compile time. Incidentally, I already wrote such a script during the last SUSE hack week a couple of weeks ago, so I'd like to share here, hopefully to give an idea for further improvements.
Quoted 8 lines
The script checks the currently loaded modules and trims other
CONFIG_XXX=msimply, and tries make oldconfig, and writes out the
resultant .config in the current directory after some checks. You can specify the config file via option, as default, it reads from/proc/config.gz.The script is VERY hackish. I should have begun with perl or whatever better script language, but I chose bash and co. So, don't expect much code quality. I'm no script guy after all :)
Check my AutoKernConf (http://cateee.net/autokernconf/), it has huge (and automatically generated) database about drivers, modules, and configuration, and it gives a minimal kernel configuration for the most of the hardware hardware.
It needs a real maintainer and tools for non-hardware drivers (fs, net, ...) and special hardware (CPU capabilities, architecture specific drivers, etc).
Quoted 5 lines
Takashi
---
Quoted 7 lines
#!/bin/sh
# this magic makes grep and co a lot faster export LANG=Ckroot=
subarch=$(uname -i)
-i is wrong (gives "unkwnown" on Debian).
Check the main Makefile how to extract the architecture from uname.
Quoted 6 lines
c) oldconfig="$OPTARG" if [ ! -f "$oldconfig" ]; then echo "Cannot find old config $oldconfig" exit 1 fi
Check also the config-* in /boot
Quoted 7 lines
find_kconfig () {
module=$1
modregex=$(echo $1 | sed -e's/_/[_-]/g')
grep -E '^[[:space:]]*obj-\$\([A-Za-z0-9_]*\)[[:space:]]*[+:]=.*[[:space:]]'"$modregex"'\.o' $tmpd/makefiles | \
sed -e's/^.*://g' -e's/^[[:space:]]*obj-\$(\(.*\)).*$/\1/g'
}My python code works better (handle "alias", etc.). Anyway I should send some cleanup patches to correct and improve some ugly kernel Makefile.
ciao
cate
At Thu, 18 Sep 2008 16:01:50 +0200,
Quoted 20 lines · Giacomo A. Catenazzi
Quoted 14 lines · Takashi Iwai
Hi,
a topic that just came up in kernel summit is a script to remove unneeded kernel configs automatically to reduce the compile time. Incidentally, I already wrote such a script during the last SUSE hack week a couple of weeks ago, so I'd like to share here, hopefully to give an idea for further improvements.
The script checks the currently loaded modules and trims other
CONFIG_XXX=msimply, and tries make oldconfig, and writes out the resultant .config in the current directory after some checks. You can specify the config file via option, as default, it reads from/proc/config.gz.Quoted 3 lines
The script is VERY hackish. I should have begun with perl or whatever better script language, but I chose bash and co. So, don't expect much code quality. I'm no script guy after all :)
Quoted 8 lines
Check my AutoKernConf (http://cateee.net/autokernconf/), it has huge (and automatically generated) database about drivers, modules, and configuration, and it gives a minimal kernel configuration for the most of the hardware hardware.
It needs a real maintainer and tools for non-hardware drivers (fs, net, ...) and special hardware (CPU capabilities, architecture specific drivers, etc).
Thanks, this looks good. Nice to see other people already working on a similar issue.
Quoted 4 lines
Quoted 1 line
subarch=$(uname -i)
-i is wrong (gives "unkwnown" on Debian).
Check the mainMakefilehow to extract the architecture from uname.
Oh yeah, another hackish thing...
Quoted 8 lines
Quoted 6 lines
c) oldconfig="$OPTARG" if [ ! -f "$oldconfig" ]; then echo "Cannot find old config $oldconfig" exit 1 fiCheck also the config-* in /boot
OK, such distro specific things have to be handled better. I didn't know of /boot/config-* at all.
OTOH, if /proc/config.gz exists, it's the right place to see because
of an obvious reason. So, checking other paths should be a kind of
fallback.
Quoted 9 lines
Quoted 7 lines
find_kconfig () { module=$1 modregex=$(echo $1 | sed -e's/_/[_-]/g') grep -E '^[[:space:]]*obj-\$\([A-Za-z0-9_]*\)[[:space:]]*[+:]=.*[[:space:]]'"$modregex"'\.o' $tmpd/makefiles | \ sed -e's/^.*://g' -e's/^[[:space:]]*obj-\$(\(.*\)).*$/\1/g' }My python code works better (handle "alias", etc.).
We don't need to handle alias because the whole module names come from lsmod output. That's so simple.
OTOH, I don't think the parsing in my script is perfect at all, though. It's a result of one or two hours hack. Improving that function a bit more robust would be pretty helpful.
Regarding the kernel config database: it's nice, indeed. But, I don't think the whole database is required for the task I tried to achieve. It wasn't intended to create a kernel config from scratch. It's not necessarily the absolute minimum. Instead, it just trims the unnecessary module builds. For that purpose, the database is not needed by assumption of the all required modules have been loaded. This is rather a safer option than creating a minimum kconfig from scratch.
thanks,
Takashi
Hi Takashi,
Quoted 8 lines · On Wed, 17 Sep 2008 01:55:26 +0200 Takashi Iwai <[email protected]> wrote:
Hi,
a topic that just came up in kernel summit is a script to remove unneeded kernel configs automatically to reduce the compile time. Incidentally, I already wrote such a script during the last SUSE hack week a couple of weeks ago, so I'd like to share here, hopefully to give an idea for further improvements.
Quoted 5 lines
The script checks the currently loaded modules and trims other
CONFIG_XXX=msimply, and tries make oldconfig, and writes out the resultant .config in the current directory after some checks. You can specify the config file via option, as default, it reads from/proc/config.gz.
I agree with Giacomo: you should also check for /boot/config-uname -r and /boot/config-uname -r.gz.
Also, if the idea is to make life easier for kernel newbies, I think the better would be to have the script called at kernel Makefile (something like "make diet"), and having a few other config's available somewhere at kernel tree, since even a "minimal" kernel may need (or not) a few random modules, like usb-storage. So, I think we should open a dialog that allows the selection of using the previous kernel config or selecting between a few profiles like "minimalist" (just the auto-detected things), "desktop" (adding things like usb-storage), "notebook" (with a power-saving optimized config), "server" (adding stuff like LVM, RAID5 and some advanced network configs).
It seems a good idea to check at /proc/cpuinfo and optimize for that specific processor, enabling SMP only if needed, and only for the number of existing cores.
Quoted 3 lines
The script is VERY hackish. I should have begun with perl or whatever better script language, but I chose bash and co. So, don't expect much code quality. I'm no script guy after all :)
Using just a shellscript and binutils seems to be better than using other tools, since it allows the usage on minimal configured systems where the user might not have perl or other scripting languages.
Cheers,
Mauro.
Quoted 5 lines · Mauro Carvalho Chehab
Hi Takashi,
On Wed, 17 Sep 2008 01:55:26 +0200
Takashi Iwai <[email protected]> wrote:
Quoted 7 lines
Quoted 3 lines
The script is VERY hackish. I should have begun with perl or whatever better script language, but I chose bash and co. So, don't expect much code quality. I'm no script guy after all :)
Using just a shellscript and binutils seems to be better than using other tools, since it allows the usage on minimal configured systems where the user might not have perl or other scripting languages.
But very slow, complex and not enough powerful:
The tools should have extensive knowledge about Kconfig, and the CONFIG_ dependencies; it should handle a lot of structures (text/binary; line/record oriented; etc.), which it is not easy in shell (awk, od, sed with extended regexp are not core utilities).
So my suggestion is: do it in a high level language, to find and stabilize the design, and then integrate it in kconfig later (using C).
ciao
cate
Quoted 10 lines · Giacomo A. Catenazzi
Quoted 9 lines · Mauro Carvalho Chehab
Hi Takashi,
Quoted 3 lines · On Wed, 17 Sep 2008 01:55:26 +0200 Takashi Iwai <[email protected]> wrote:
The script is VERY hackish. I should have begun with perl or whatever better script language, but I chose bash and co. So, don't expect much code quality. I'm no script guy after all :)
Quoted 5 lines
Quoted 3 lines
Using just a shellscript and binutils seems to be better than using other tools, since it allows the usage on minimal configured systems where the user might not have perl or other scripting languages.
But very slow,
I don't think that performance is a big issue here. I tested Takashi scripts on this old 1.5 GHz centrino one core notebook, and it runned in a few seconds. Kernel compilation with diet config took 30-40 mins. The additional time for the script didn't significantly increased the total time.
Quoted 1 line
complex and not enough powerful:
For sure a scripting language like perl is more appropriate.
Quoted 2 lines
The tools should have extensive knowledge about
Kconfig, and theCONFIG_dependencies
Since CONFIG_ and their dependencies are constantly changing/adding/removing at kernel, the better would be to have a good Kconfig parsing, avoiding to have lots of "local knowledge" about symbols. The better model seems to be the one where subsystem maintainers don't need to take extra care for not breaking the tool when touching at Kconfig/Makefiles.
Also, it occured me that some drivers have "implicit" Kconfig dependencies inside that may limit its work. Looking on a random driver, just as an example, you have things like:
$ grep ifdef cpufreq_stats.c cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS
So, I suspect that the script might also need to parse the selected files to seek for additional rules.
I'm not sure what to do after discovering such things: Assume the option that generates smaller code? Assume the option that generates the most complete code? Maybe the better would be to use the selected config profile as a hint for such symbols. If not found there, then ask user.
Quoted 4 lines
it should handle a lot of
structures (text/binary; line/record oriented; etc.), which it is not easy in shell (awk, od, sed with extended regexp are not core utilities).
True.
Quoted 3 lines
So my suggestion is: do it in a high level language, to find and stabilize the design, and then integrate it in kconfig later (using C).
Seems a good approach to me.
Signature · 2 lines
Cheers, Mauro
Instead of replying all other emails, it is simpler to explain my approach.
The purpose of my program is to find all CONFIG_s, which are needed to support all the hardware (and some more), and to show them in menuconfig. This is the main difference with Takashi purpose:
he wants to build a bootable (without initramfs) kernel, starting
to an high modular kernel (usually found in distribution),
so no usable on an already optimized machine.
Anyway there are a lot of overlapping tasks, which we could exchange.
My design (files and sample output are in http://cateee.net/sources/autokernconf/autokernconf-2008-08-29/ ):
kdetect.sh: detects hardwares, in a redundant manner, in order to have data if there are missing one ore more between: /proc, /sys, lspci, ...
autokernconf.sh: a shell script which read the driver database (lkddb.list) and kdetect output; it generates the desired CONFIG_s.
So far, so simple, and similar to Takashi method. I only split the task in two programs (detection and configuration).
The big difference is in the additional program, which generated the
driver database: it reads all sources, it checks 25 subsystems
(see http://cateee.net/lkddb/statistics.html ), taking in account
to preprocessors (#ifdef CONFIG_...), #defines, static strings, masks, all architectures, etc. Additionally it parse Makefile and Kconfig to find the dependencies
of every source file.
It misses some generic detection (CPU, memory), which Takashi is doing, and the kconfig integration.
So I think most part could be integrated with the Takashi proposal, and doing optionally the parsing of full driver database.
ciao
cate
Quoted 34 lines · Mauro Carvalho Chehab
Quoted 2 lines · Giacomo A. Catenazzi
The tools should have extensive knowledge about
Kconfig, and theCONFIG_dependenciesSince
CONFIG_and their dependencies are constantly changing/adding/removing at kernel, the better would be to have a goodKconfigparsing, avoiding to have lots of "local knowledge" about symbols. The better model seems to be the one where subsystem maintainers don't need to take extra care for not breaking the tool when touching at Kconfig/Makefiles.Also, it occured me that some drivers have "implicit"
Kconfigdependencies inside that may limit its work. Looking on a random driver, just as an example, you have things like:$ grep ifdef cpufreq_stats.c cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILS cpufreq_stats.c:#ifdef CONFIG_CPU_FREQ_STAT_DETAILSSo, I suspect that the script might also need to parse the selected files to seek for additional rules.
I'm not sure what to do after discovering such things: Assume the option that generates smaller code? Assume the option that generates the most complete code? Maybe the better would be to use the selected config profile as a hint for such symbols. If not found there, then ask user.
Quoted 4 lines
it should handle a lot of
structures (text/binary; line/record oriented; etc.), which it is not easy in shell (awk, od, sed with extended regexp are not core utilities).True.
Quoted 3 lines
So my suggestion is: do it in a high level language, to find and stabilize the design, and then integrate it in kconfig later (using C).
Seems a good approach to me.
At Fri, 19 Sep 2008 11:12:34 +0200,
Quoted 10 lines · Giacomo A. Catenazzi
Instead of replying all other emails, it is simpler to explain my approach.
The purpose of my program is to find all
CONFIG_s, which are needed to support all the hardware (and some more), and to show them in menuconfig. This is the main difference with Takashi purpose:
he wants to build a bootable (without initramfs) kernel, starting to an high modular kernel (usually found in distribution), so no usable on an already optimized machine.
Yes. Your stuff would be actually helpful for people who want a really slim kernel config.
Quoted 2 lines
Anyway there are a lot of overlapping tasks, which we could exchange.
Right, for example, kconfig parser can be shared. Mine is pretty simplistic, and very likely incomplete.
Quoted 21 lines
My design (files and sample output are in
http://cateee.net/sources/autokernconf/autokernconf-2008-08-29/ ):
kdetect.sh: detects hardwares, in a redundant manner, in order to have data if there are missing one ore more between: /proc, /sys, lspci, ...
autokernconf.sh: a shell script which read the driver database (lkddb.list) and kdetect output; it generates the desiredCONFIG_s.So far, so simple, and similar to Takashi method. I only split the task in two programs (detection and configuration).
The big difference is in the additional program, which generated the driver database: it reads all sources, it checks 25 subsystems (see http://cateee.net/lkddb/statistics.html ), taking in account to preprocessors (#ifdef
CONFIG_...), #defines, static strings, masks, all architectures, etc.
Additionally it parseMakefileandKconfigto find the dependencies of every source file.
Quoted 5 lines
It misses some generic detection (CPU, memory), which Takashi is doing, and the kconfig integration.
So I think most part could be integrated with the Takashi proposal, and doing optionally the parsing of full driver database.
Agreed.
I think a good start would be to integrate a simple version of parser and trim-down mechanism into the upstream kbuild system, so that you can call such as "make diet", as Mauro mentioned.
BTW, Steve Rostedt told in the last night that he uses also his own stuff for the same diet kconfig.
Steve, can we share your script, too?
thanks,
Takashi
Quoted 2 lines · Takashi Iwai
Yes. Your stuff would be actually helpful for people who want a really slim kernel config.
Incidentally, I wrote such a script as well. In 64 lines of python code. I attach the script in question in the mail. I use a approach very similar to yours. But I work around some of the nasty part.
It needs to start with kernel config has most of the modules enabled. Most of the kernel config from the linux distribution is like that. Then it try to parse the Kbuild looking for what kernel config produce what module. It is not a 1 to 1 mapping.
Once we have the mapping. It open /proc/modules to go over every module. Remove the config which can produce those module from the mapping.
Now we have a blacklist of config produce modules but not the one we want.
Then the last stage is just open a config file and filter out the config option on the blacklist. Write the result to a new config file.
I avoid arch/ and firmware/ because it is nasty. I figure module in arch/ is small enough I don't mind building it. And the firmware directory, if I don't build the module loads it. Those firmware will automatically skipped any way.
The little tricky part is some thing like this:
========================
ifeq ($(CONFIG_BLK_DEV_CMD640), m)
obj-m += cmd640.o
endif
========================
Internally it get convert it into:
"obj-$(CONFIG_BLK_DEV_CMD640) += cmd640.o"
For easier parsing.
It is pretty fast as well. Less then a second if the file cache is hot.
$ time minmod.py .config reading Kbuild Makefile 593 files acpi_cpufreq : Uknown module, may be firmware? dcdbas : Uknown module, may be firmware? 1560 module option disabled Writing new config to: .config.min
real 0m0.470s
user 0m0.303s
sys 0m0.075s
$
Chris
Quoted 3 lines · Chris Li
Incidentally, I wrote such a script as well. In 64 lines of python code. I attach the script in question in the mail. I use a approach very similar to yours. But I work around some of the nasty part.
Oops, I forget to attach the script. Here it is.
Chris
#!/usr/bin/python
import re
import os
class Kbuild:
def __init__(self, root):
self.root = root
self.confs = {} # dict of lists
self.modules = {}
self.read_kbuild()
def all_kbuild(self):
cmd = 'find . '\
' -regex "%s/[a-z].*/\\(Makefile\\|Kbuild\\)"' \
' -not -regex ".*/\(' \
'arch\|Documents\|firmware\|scripts\|raid6test\)/.*"' \
% self.root
return os.popen(cmd).read().split()
def read_kbuild(self):
allfile = [open(f).read() for f in self.all_kbuild()]
data = "\n".join(allfile)
data = re.sub(r"(?m)^ifeq\s+\(\$\((CONFIG_\S+)\s*\),\s*m\).*",
r"obj-$(\1) \\", data)
data = re.sub(r"(?m)\\\n\s*|obj-m ", '', data)
all = re.findall( r"(?m)obj-\$\((CONFIG_\S+)\)\s*[:+]=(.*)", data)
self.data = data
for (conf, modules) in all:
mods = re.findall(r"(\S+)\.o", modules)
for m in mods:
m = re.sub("-","_",m)
#print conf, "->", m
self.confs.setdefault(conf, []).append(m)
self.modules.setdefault(m, []).append(conf)
def read_modules(self):
data = open("/proc/modules").read()
return re.findall(r"(?m)^\S+", data)
def test_module(self, m):
all, conf = m.group(0,1)
if conf in self.confs:
return "# %s is not set"%conf
return all
def filter(self, config):
#print self.data
for m in self.read_modules():
if not self.modules.has_key(m):
print m,": Uknown module, may be firemware?"
continue
for c in self.modules[m]:
if c in self.confs:
del self.confs[c]
data = open(config).read()
data = re.sub(r"(?m)^(CONFIG_\S+)=m", self.test_module, data)
out = config + ".min"
print "Writing new config to:", out
open(out, "w").write(data)def main():
kbuild = Kbuild(".")
kbuild.filter(".config")main()
At Fri, 19 Sep 2008 16:55:51 -0700,
Quoted 8 lines · Chris Li
Quoted 2 lines · Takashi Iwai
Yes. Your stuff would be actually helpful for people who want a really slim kernel config.
Incidentally, I wrote such a script as well. In 64 lines of python code. I attach the script in question in the mail. I use a approach very similar to yours. But I work around some of the nasty part.
Thanks, I'll check yours too soon.
Quoted 5 lines
It needs to start with kernel config has most of the modules enabled. Most of the kernel config from the linux distribution is like that. Then it try to parse the
Kbuildlooking for what kernel config produce what module. It is not a 1 to 1 mapping.
Quoted 8 lines
Once we have the mapping. It open
/proc/modulesto go over every module. Remove the config which can produce those module from the mapping.Now we have a blacklist of config produce modules but not the one we want.
Then the last stage is just open a config file and filter out the config option on the blacklist. Write the result to a new config file.
I think a whitelist would be useful, too, e.g. for hotplug devices like usb.
Quoted 4 lines
I avoid
arch/andfirmware/because it is nasty. I figure module inarch/is small enough I don't mind building it. And the firmware directory, if I don't build the module loads it. Those firmware will automatically skipped any way.
Agreed.
Quoted 8 lines
The little tricky part is some thing like this:
========================
ifeq ($(CONFIG_BLK_DEV_CMD640), m)obj-m += cmd640.oendif
========================
Internally it get convert it into:
"obj-$(CONFIG_BLK_DEV_CMD640) += cmd640.o"
This should be fixed in Makefile.
Care to submit a patch?
Takashi
Quoted 18 lines · Takashi Iwai
At Fri, 19 Sep 2008 16:55:51 -0700,
Quoted 4 lines · Chris Li
I avoid
arch/andfirmware/because it is nasty. I figure module inarch/is small enough I don't mind building it. And the firmware directory, if I don't build the module loads it. Those firmware will automatically skipped any way.Agreed.
Quoted 8 lines
The little tricky part is some thing like this:
========================
ifeq ($(CONFIG_BLK_DEV_CMD640), m)obj-m += cmd640.oendif
========================
Internally it get convert it into:
"obj-$(CONFIG_BLK_DEV_CMD640) += cmd640.o"
Are you sure?
I.e. if you set CONFIG_BLK_DEV_CMD640=y, you see that the two constructs gives different results.
If the makefile author did such complex construct, I really think there is a reason ;-)
ciao
cate
Quoted 11 lines
This should be fixed in
Makefile.
Care to submit a patch?Takashi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 22 lines · Giacomo A. Catenazzi
Quoted 20 lines · Takashi Iwai
At Fri, 19 Sep 2008 16:55:51 -0700,
Quoted 6 lines · Chris Li
I avoid
arch/andfirmware/because it is nasty. I figure module inarch/is small enough I don't mind building it. And the firmware directory,
if I don't build the module loads it. Those firmware will automatically skipped
any way.Agreed.
Quoted 8 lines
The little tricky part is some thing like this:
========================
ifeq ($(CONFIG_BLK_DEV_CMD640), m)obj-m += cmd640.oendif
========================
Internally it get convert it into:
"obj-$(CONFIG_BLK_DEV_CMD640) += cmd640.o"
Quoted 7 lines
Are you sure?
I.e. if you setCONFIG_BLK_DEV_CMD640=y, you see that the two constructs gives different results.If the makefile author did such complex construct, I really think there is a reason ;-)
Yes indeed, we should find out what the author meant.
ifeq ($(CONFIG_BLK_DEV_CMD640), m)
obj-m += cmd640.o
endif
and
obj-$(CONFIG_BLK_DEV_CMD640) += cmd640.o
mean two different things.
The former means to compile the cmd640 only if it is set to a module. The later means to compile the cmd640 if it is either a module or built into the core kernel code.
The author needs to make this change or explain why it is like this.
-- Steve
At Mon, 22 Sep 2008 11:41:36 -0400 (EDT),
Quoted 42 lines · Steven Rostedt
Quoted 22 lines · Giacomo A. Catenazzi
Quoted 20 lines · Takashi Iwai
At Fri, 19 Sep 2008 16:55:51 -0700,
Quoted 6 lines · Chris Li
I avoid
arch/andfirmware/because it is nasty. I figure module inarch/is small enough I don't mind building it. And the firmware directory,
if I don't build the module loads it. Those firmware will automatically skipped
any way.Agreed.
Quoted 8 lines
The little tricky part is some thing like this:
========================
ifeq ($(CONFIG_BLK_DEV_CMD640), m)obj-m += cmd640.oendif
========================
Internally it get convert it into:
"obj-$(CONFIG_BLK_DEV_CMD640) += cmd640.o"Quoted 7 lines
Are you sure?
I.e. if you setCONFIG_BLK_DEV_CMD640=y, you see that the two constructs gives different results.If the makefile author did such complex construct, I really think there is a reason ;-)
Yes indeed, we should find out what the author meant.
ifeq ($(CONFIG_BLK_DEV_CMD640), m)
obj-m += cmd640.oendif
and
Quoted 7 lines
obj-$(CONFIG_BLK_DEV_CMD640) += cmd640.o
mean two different things.
The former means to compile the cmd640 only if it is set to a module. The later means to compile the cmd640 if it is either a module or built into the core kernel code.
FYI, in this particular case, cmd640.o is linked in the upper Makefile for CONFIG_BLK_DEV_CMD640=y.
Quoted 1 line
The author needs to make this change or explain why it is like this.
Right. My guess is the link order, but still there must be a better way to fix.
thanks,
Takashi
Quoted 2 lines · Takashi Iwai
I think a whitelist would be useful, too, e.g. for hotplug devices like usb.
The white list is marginally useful. But it is very complicate to do. In order to preform white list. You have to know the dependency of the module. e.g. You need to know exactly which options to turn on so the modules will show up. The current dependency analyze is the other way around. It knows which device should show up when some config option is enabled. Answering the question of which option should enable in order that device show up is a much harder problem.
Also, as long as the user can get a boot able kernel out of the minimal config. It is easy enough to recompile the kernel with white list modules and install them as needed.
The white list should be a secondary step to do because its complexity. The first priority is to have some thing simple and provide a good minimal kernel module option. Then we can go from there, how to add extra white list from it. The white list is a very different game any way. I found the minimal config option is a sweet spot to have in terms of feature and simplicity.
Quoted 2 lines
This should be fixed in
Makefile.
Care to submit a patch?
Some module does not want to be compile as build-in.
Chris
At Mon, 22 Sep 2008 12:25:21 -0700,
Quoted 6 lines · Chris Li
Quoted 2 lines · Takashi Iwai
I think a whitelist would be useful, too, e.g. for hotplug devices like usb.
Quoted 8 lines
The white list is marginally useful. But it is very complicate to do. In order to preform white list. You have to know the dependency of the module. e.g. You need to know exactly which options to turn on so the modules will show up. The current dependency analyze is the other way around. It knows which device should show up when some config option is enabled. Answering the question of which option should enable in order that device show up is a much harder problem.
That is true. Enabling arbitrary kernel config item is hard to achieve right now. I think this feature should be implemented in kbuild parser itself. The current reverse-select is way limited and known to be problematic in many kernel configs.
Anyway, I think the white list isn't that hard for certain use-cases. My assumption is that users base on the distro kernel that have already all modules. Then, for likely scenarios such as USB-storage hotplug, we can simply provide possible modules such as usb-storage and nls_* as the additional modules. The script would just need to add them to the existing module list. If necessary, the module dependency can be solved easily from modules.dep, too.
Quoted 3 lines
Also, as long as the user can get a boot able kernel out of the minimal config. It is easy enough to recompile the kernel with white list modules and install them as needed.
I don't think choosing necessary kernel config is "easy enough". If it were easy enough, we don't need such scripts at all. The problem is that you must know exactly which config should be used for what purpose. This is often a high hurdle.
Quoted 6 lines
The white list should be a secondary step to do because its complexity. The first priority is to have some thing simple and provide a good minimal kernel module option. Then we can go from there, how to add extra white list from it. The white list is a very different game any way. I found the minimal config option is a sweet spot to have in terms of feature and simplicity.
I feel you goal is much farther than mine :)
The white list can be just a list of modules. If it doesn't work, that's fine -- at least, we get the exactly same system like currently running.
Quoted 4 lines
Quoted 2 lines
This should be fixed in
Makefile.
Care to submit a patch?Some module does not want to be compile as build-in.
But not about this case. It wants to be compiled-in, too.
thanks,
Takashi
Quoted 4 lines · Takashi Iwai
That is true. Enabling arbitrary kernel config item is hard to achieve right now. I think this feature should be implemented in kbuild parser itself. The current reverse-select is way limited and known to be problematic in many kernel configs.
I was looking at the kbuild system. BTW, I really like the Makefile in kbuild. For the reverse select, here is what I have in mind.
The reverse select needs to maintain the define-user chain for each kernel option. And each kernel option has a list of the kernel option and value pare to enable an option. Once we produce such an list for "all config". We can know exactly what kernel option needs to set in order to get to so module. It is kind of doing the data flow analyze on the config options.
It is not trivial, but it shouldn't be too hard either.
Quoted 7 lines
Anyway, I think the white list isn't that hard for certain use-cases. My assumption is that users base on the distro kernel that have already all modules. Then, for likely scenarios such as USB-storage hotplug, we can simply provide possible modules such as usb-storage and nls_* as the additional modules. The script would just need to add them to the existing module list. If necessary, the module dependency can be solved easily from modules.dep, too.
If you just want a white list without resolving the dependency, that should be very easy to do in my current minmod.py.
In terms of implement in C. The currently kconf symbol has most of the bits ready already. We can add a symbol type "module". It points to a list of config symbols. That should be good enough. Then we can use 1 bit of flag in config symbol to mark it is blacklisted or not.
We can use some thing like minmod.py to provide a prototype for some kernel user to try it out. If it covers most of the feature the we want. We can go ahead to the C version.
Quoted 1 line
But not about this case. It wants to be compiled-in, too.
I think my script can catch those already. Maybe I will beat it up to generate a list of similar configs using "obj-m" directly. We can example them one by one.
Chris
At Tue, 23 Sep 2008 11:15:19 -0700,
Quoted 10 lines · Chris Li
Quoted 4 lines · Takashi Iwai
That is true. Enabling arbitrary kernel config item is hard to achieve right now. I think this feature should be implemented in kbuild parser itself. The current reverse-select is way limited and known to be problematic in many kernel configs.
I was looking at the kbuild system. BTW, I really like the
Makefilein kbuild. For the reverse select, here is what I have in mind.
Quoted 8 lines
The reverse select needs to maintain the define-user chain for each kernel option. And each kernel option has a list of the kernel option and value pare to enable an option. Once we produce such an list for "all config". We can know exactly what kernel option needs to set in order to get to so module. It is kind of doing the data flow analyze on the config options.
It is not trivial, but it shouldn't be too hard either.
Well, it is not so easy, too.
The good-working reverse selection with dependency checks is a feature we've long for in Kconfig, but it hasn't been implemented because it could easily introduce a messy dependency loop, or some cases are not easily resolvable.
Anyway, the development of reverse-selection should involve with Sam and other Kbuild maintainer/developers.
Takashi
Quoted 5 lines · Takashi Iwai
Well, it is not so easy, too.
The good-working reverse selection with dependency checks is a feature we've long for inKconfig, but it hasn't been implemented because it could easily introduce a messy dependency loop, or some cases are not easily resolvable.
What I describe is a simpler version of that. The key difference is that, I only work out the dependency base on a distro config file. So there is only one set of config value to generate a module instead of many combo in the more general case. I think that will cover 80 percent of what user want. If you have a known all config file, the reverse selection is easy resolvable.
Quoted 3 lines
Anyway, the development of reverse-selection should involve with Sam and other
Kbuildmaintainer/developers.
Care to CC them? I think most likely you need to touch the current kconf for "make diet" any way.
Thanks
Chris
At Thu, 18 Sep 2008 17:43:48 +0200,
Quoted 15 lines · Giacomo A. Catenazzi
Quoted 5 lines · Mauro Carvalho Chehab
Hi Takashi,
On Wed, 17 Sep 2008 01:55:26 +0200
Takashi Iwai <[email protected]> wrote:Quoted 7 lines
Quoted 3 lines
The script is VERY hackish. I should have begun with perl or whatever better script language, but I chose bash and co. So, don't expect much code quality. I'm no script guy after all :)
Using just a shellscript and binutils seems to be better than using other tools, since it allows the usage on minimal configured systems where the user might not have perl or other scripting languages.
Quoted 6 lines
But very slow, complex and not enough powerful:
The tools should have extensive knowledge aboutKconfig, and theCONFIG_dependencies; it should handle a lot of structures (text/binary; line/record oriented; etc.), which it is not easy in shell (awk, od, sed with extended regexp are not core utilities).
It's not that slow.
Quoted 3 lines
So my suggestion is: do it in a high level language, to find and stabilize the design, and then integrate it in kconfig later (using C).
Maybe a good idea to add such functionality into kbuild itself. It's THE parser after all.
thanks,
Takashi
At Thu, 18 Sep 2008 12:25:41 -0300,
Quoted 14 lines · Mauro Carvalho Chehab
Hi Takashi,
Quoted 8 lines · On Wed, 17 Sep 2008 01:55:26 +0200 Takashi Iwai <[email protected]> wrote:
Hi,
a topic that just came up in kernel summit is a script to remove unneeded kernel configs automatically to reduce the compile time. Incidentally, I already wrote such a script during the last SUSE hack week a couple of weeks ago, so I'd like to share here, hopefully to give an idea for further improvements.
Quoted 8 lines
Quoted 5 lines
The script checks the currently loaded modules and trims other
CONFIG_XXX=msimply, and tries make oldconfig, and writes out the resultant .config in the current directory after some checks. You can specify the config file via option, as default, it reads from/proc/config.gz.I agree with Giacomo: you should also check for
/boot/config-uname -rand/boot/config-uname -r.gz.
OK, I'll add the check of these as fallback.
Any known paths for other distros?
Quoted 9 lines
Also, if the idea is to make life easier for kernel newbies, I think the better would be to have the script called at kernel
Makefile(something like "make diet"), and having a few other config's available somewhere at kernel tree, since even a "minimal" kernel may need (or not) a few random modules, like usb-storage. So, I think we should open a dialog that allows the selection of using the previous kernel config or selecting between a few profiles like "minimalist" (just the auto-detected things), "desktop" (adding things like usb-storage), "notebook" (with a power-saving optimized config), "server" (adding stuff like LVM, RAID5 and some advanced network configs).
There are different perspectives for this. My goal to reduce the build time with the existing kernel config for a specific hardware. That is, it makes eaiser to create another set of kernel with test patches or bisect.
OTOH, it would be definitely useful if we can create a clean minimum kernel config from scratch based on the hardware information. For that, the use-case specific configuration (or preset) would be good.
Quoted 7 lines
Quoted 3 lines
The script is VERY hackish. I should have begun with perl or whatever better script language, but I chose bash and co. So, don't expect much code quality. I'm no script guy after all :)
Using just a shellscript and binutils seems to be better than using other tools, since it allows the usage on minimal configured systems where the user might not have perl or other scripting languages.
Well, perl or python is most likely installed on every modern system, so I think we don't stick with bash & co. But, I see your point, and it's also the reason I started with bash, etc.
thanks,
Takashi
Quoted 4 lines · Takashi Iwai
There are different perspectives for this. My goal to reduce the build time with the existing kernel config for a specific hardware. That is, it makes eaiser to create another set of kernel with test patches or bisect.
One has to be careful here - I've more than once hit kernel bugs that I ended up bisecting, which were dependent on the exact .config used. The most recent was a deadlock during early boot if CONFIG_HID_COMPAT=y, which would have been turned off by any sane kconfig trimmer. On the flip side, running a trimmer wouldn't minimize my .config very much, because I've already set most stuff I *never* use to 'n'.
And more than once, I've had a bisect go horribly awry and need to be re-started because the bisect crossed back and forth over a commit that added a config option, so doing a 'make oldconfig' at each iteration would re-prompt, and I wasn't consistent in the reply I gave. Usually doesn't matter, except when the broken code is in the support for the feature...
I *do* keep around a *very* minimal "only what I need to get to single-user" config if build time is an issue...
Might be different if you're starting off with a distro config that's basically an 'allmodconfig' - lotta fat to trim off *that*....
At Thu, 18 Sep 2008 17:09:27 -0400,
Quoted 10 lines · [email protected] wrote:
Quoted 4 lines · Takashi Iwai
There are different perspectives for this. My goal to reduce the build time with the existing kernel config for a specific hardware. That is, it makes eaiser to create another set of kernel with test patches or bisect.
One has to be careful here - I've more than once hit kernel bugs that I ended up bisecting, which were dependent on the exact .config used.
That's true. But, you should at least test the trimmed kernel once before bisecting to reproduce the bug.
And, my script doesn't do anything so intrusive. It trims just unused CONFIG_*=m. Since they are not loaded anyway, reducing them must be harmless (in theory).
Quoted 5 lines
The most recent
was a deadlock during early boot ifCONFIG_HID_COMPAT=y, which would have been turned off by any sane kconfig trimmer. On the flip side, running a trimmer wouldn't minimize my .config very much, because I've already set most stuff I *never* use to 'n'.
Heh, you are obviously not a "typical" user ;)
My scenario was for users of distro kernel who want to help debugging with some given patches or bisecting.
Quoted 5 lines
And more than once, I've had a bisect go horribly awry and need to be re-started because the bisect crossed back and forth over a commit that added a config option, so doing a 'make oldconfig' at each iteration would re-prompt, and I wasn't consistent in the reply I gave. Usually doesn't matter, except when the broken code is in the support for the feature...
Bisect isn't perfect, indeed. But, in many cases, it can catch the harmful commit more easily, especially if it's in the area where you are unfamiliar.
Quoted 5 lines
I *do* keep around a *very* minimal "only what I need to get to single-user" config if build time is an issue...
Might be different if you're starting off with a distro config that's basically an 'allmodconfig' - lotta fat to trim off *that*....
Exactly.
thanks,
Takashi