Lines for locations sizing information (from Kernel Summit 2012 discussion)
Because I had not done any size profiling since the original RFC posting, I took 20 min and ported the patch set up to the 3.6-rc3 and booted the kernel in each configuration. There are still probably a few use cases where this patch set might be useful but the general sentiment seemed that this feature it is probably a bit too wasteful.
The 20 megs that I mentioned yesterday at the kernel summit is not the actual size taken in ram or the compressed boot image (more on that later). The 20 megs was the raw size of the ascii text file that readelf emitted from processing the vmlinux file. This file is used to construct the hash table used by kallsyms via a slightly modified scripts/kallsyms.
Example:
% ./scripts/readelf --debug-dump=line ./vmlinux -q > decoded_lines % ls -l decoded_lines |awk '{print $5" "$9}'
20312068 decoded_lines
Here is the data for the raw sizing in bytes for the exact same kernel.
4347104 - bzImage.no_lines_for_locations
6573280 - bzImage.lines_for_locations
Delta: 2,226,176
For a kernel module things are a bit different. The kernel modules are actually no different in size with or without this patch. The difference comes at the time kernel module is loaded. Instead of the in kernel module load routine throwing away the lines for locations section, it reads it and constructs a hash table which will be read at a later point.
Taking the data from readelf for my example kernel module looks like:
[22] .debug_line PROGBITS 0000000000000000 0000e143
0000000000000bcd 0000000000000000 0 0 1
Prior to hash construction we are talking about 56K, and post constructed hash for this one in kernel memory is ~30K.
It is worth pointing out that you could use this just for 1 or more kernel modules and not the core kernel since the tables are completely separate in the kernel memory, much like the kallsyms data is separated for modules and the core kernel. Just because you use one, doesn't mean you have to use the other, in the case of the lines for locations.
Here are the statistics for the in-kernel memory footprint.
On the left is the data from /proc/meminfo on the non-lines-for-locations kernel immediately after the system has booted. On the right is the same data but for the lines-for-locations built into the kallsyms table.
MemTotal: 366816 kB MemTotal: 376804 kB MemFree: 355596 kB MemFree: 365580 kB Buffers: 0 kB Buffers: 0 kB Cached: 2640 kB Cached: 2640 kB SwapCached: 0 kB SwapCached: 0 kB Active: 1256 kB Active: 1268 kB Inactive: 1936 kB Inactive: 1936 kB Active(anon): 552 kB Active(anon): 564 kB Inactive(anon): 0 kB Inactive(anon): 0 kB Active(file): 704 kB Active(file): 704 kB Inactive(file): 1936 kB Inactive(file): 1936 kB Unevictable: 0 kB Unevictable: 0 kB Mlocked: 0 kB Mlocked: 0 kB SwapTotal: 0 kB SwapTotal: 0 kB SwapFree: 0 kB SwapFree: 0 kB Dirty: 0 kB Dirty: 0 kB Writeback: 0 kB Writeback: 0 kB AnonPages: 556 kB AnonPages: 572 kB Mapped: 860 kB Mapped: 852 kB Shmem: 0 kB Shmem: 0 kB Slab: 4524 kB Slab: 4532 kB SReclaimable: 1676 kB SReclaimable: 1676 kB SUnreclaim: 2848 kB SUnreclaim: 2856 kB KernelStack: 288 kB KernelStack: 288 kB PageTables: 76 kB PageTables: 80 kB NFS_Unstable: 0 kB NFS_Unstable: 0 kB Bounce: 0 kB Bounce: 0 kB WritebackTmp: 0 kB WritebackTmp: 0 kB CommitLimit: 183408 kB CommitLimit: 188400 kB Committed_AS: 10428 kB Committed_AS: 10444 kB VmallocTotal: 34359738367 kB VmallocTotal: 34359738367 kB VmallocUsed: 1352 kB VmallocUsed: 1352 kB VmallocChunk: 34359737007 kB VmallocChunk: 34359737007 kB
So the difference is about 10 megs of memory you don't get back, which is less than the 16 megs that HPA quoted for the use of kexec. :-)
---
Q: Can we do better? A: Probably
It seems like the kallsyms approach here was not as efficient as the module hash function, because kallsyms really wasn't designed to be used this way. We can probably consider using the same logic that is used to construct the module hash table. The decoded_lines file looks like this inside and has 612886 total lines in the file:
head64.c:40 0xffffffff824b1358
pgtable_64.h:110 0xffffffff824b135a
bitops.h:321 0xffffffff824b1368
tlbflush.h:51 0xffffffff824b136f
irqflags.h:20 0xffffffff824b1374
There are a only 1687 unique file entries in the example kernel:
% cat decoded_lines |sed -e 's/:.*//'|sort -u > files % ls --block-size=1 -s files
18282 files
% wc -l files
1687 files
So if we don't care about speed and don't index the file list table we would use ~18K for just the file list table. For this 64bit kernel the base offset is always 0xffffffff so we could use a 32 bit number to represent it (4 bytes). The largest file in this kernel was "tg3.c:9998", so we can use 2 bytes for the line location, and 2 bytes for the index into the stored file block. Using a bit more math here that theoretical minimum storage using a simple hash should be:
Size of array * (4 + 2 + 2):
612886 * (4 + 2 + 2) = 4903088
We add to that our 18K for the ascii file table and get ~5megs. So it would certainly be possible to cut the usage in 1/2 vs the current patch, but it still isn't < 1 meg that Linus said was the upper limit. :-)
---
Certainly there are other options of course like:
- Perhaps it is more reasonable to just use the lines for locations for the kernel modules you care about.
- Another possibility here would be to allow the kernel to use an early boot argument to control the loading of the extra data into the kallsyms. The idea being that you could selectively elect to boot with the feature on or off at runtime, if we assume disk space is relatively cheap for the bzImage storage.
- Yet one more possibility is to allow loading the extra data after the system has booted from user space.
This all started out as an experiment just to see what it would take to get lines for locations information in the kernel, because I wanted it for doing fast back traces in kdb as well as finding out what line I was on without the need for any kind of scripting. I don't know yet if it is the end of the road for these patches, but for anyone who wanted to try the latest version of this patch set it is at:
git://git.kernel.org/pub/scm/linux/kernel/git/jwessel/kgdb.git lines_for_locations_3_6
And thanks to everyone for the good discussion.
Cheers,
Jason.