Skip to contentSkip to conversation

Recent change to kernel spikes out ccache/distcc

REGRESSION Theodore Ts'o 10 participants 37 messages Jan 6, 2009 – Jan 15, 2009 Kbuild

lore
Share & tools
lore
#0 lore raw

The following commit, ad7a953c, "kbuild: strip generated symbols from *.ko" which was merged to the mainline last week (December 28th) has the unfortunate side effct of making ccache useless. That's because all files are now built using a two step process. First they are compiled to assembly via "gcc -S -o .tmp_foo.s" and then assembled via "gcc" in a separate step. Unfortunately, ccache doesn't seem to be able to cache either the compilation or the assembly step (and it may not be worth it once the two steps are separated).

It would be nice if there was a CONFIG option which compiles files the old-fashioned way, even if we lose the ability to strip the__crc_ symbols, just so that ccache can function again, since it can really speed up the kernel development process. Alternatively, maybe ccache could somehow be taught a specialized way take the .c file, some kind of kbuild-specified version identifier, and then output of the gcc -S, assembly frobnication, gcc -c, objcopy pipeline that kbuild uses to build an object file, so we can have cache the entire kbuild object building process.

Or, if that's too complicated, maybe it would be worthwhile to have kbuild create its own specialized ccache system? Note that the last two solutions rule out using distcc, unless we can encapsulate the build process from a series of Makefile macros to a shell or C program, which could then be injected to the remote host system to be executed by distcc. One value of doing that is the CRC or MD5 of the shell script could be used as the version tag for the cache system.

In the short term, though, it would be nice if we could get back a simple way of making a kernel object file using just cc, so that ccache
and distcc could be functional again. Does that seem reasonable?

Thanks,

- Ted

#1 lore raw
Quoted 4 lines
Quoted 1 line
Quoted 1 line

"Theodore Ts'o" <[email protected]> 06.01.09 16:15 >>>

In the short term, though, it would be nice if we could get back a simple way of making a kernel object file using just cc, so that ccache
and distcc could be functional again. Does that seem reasonable?

Making the new logic dependent on a config option would seem reasonable to me - of course at the expense of the respective Makefile becoming even less readable.

Jan

#2 lore raw
Quoted 8 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line

"Theodore Ts'o" <[email protected]> 06.01.09 16:15 >>>

In the short term, though, it would be nice if we could get back a simple way of making a kernel object file using just cc, so that ccache
and distcc could be functional again. Does that seem reasonable?

Making the new logic dependent on a config option would seem reasonable to me - of course at the expense of the respective Makefile becoming even less readable.

Too late. :-) It's pretty unreadable already.... as a result, I'm not at all confident that I could make such a patch. Is this something you could perhaps whip up? I'd really appreicate it, as it would seriously speed up by kernel development efforts.

Regards,

- Ted

#3 lore raw
Quoted 11 lines · Theodore Tso
Quoted 8 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line

"Theodore Ts'o" <[email protected]> 06.01.09 16:15 >>>

In the short term, though, it would be nice if we could get back a simple way of making a kernel object file using just cc, so that ccache
and distcc could be functional again. Does that seem reasonable?

Making the new logic dependent on a config option would seem reasonable to me - of course at the expense of the respective Makefile becoming even less readable.

Too late. :-) It's pretty unreadable already....

I'm all ears for inputs how to make them even a little more readable. We actually used several iteratiosn of Jan's patch just to improve readability in areas the patch touched.

I think at least half of the unreadability comes from the fact that people do not realise the steps needed to actually build the kernel.

It is a little bit more involved than doing a simple sequence of gcc; ld.

We have btw. 1140 Makefiles in the kernel of a total of 28000 lines.

Yes - exactly 28000 lines as reported by:
find -name 'Makefile*' | xargs cat | wc -l

But numbers aside - any inputs appreciated.

Sam

#4 lore raw
Quoted 7 lines · Sam Ravnborg

I'm all ears for inputs how to make them even a little more readable. We actually used several iteratiosn of Jan's patch just to improve readability in areas the patch touched.

I think at least half of the unreadability comes from the fact that people do not realise the steps needed to actually build the kernel.

Yeah, I thought about that a lot, but I couldn't come up with a constructive suggestion. The best I could come up with to have kbuild generate a shell script helper or use a shell script helper to actually build each .o file. However, the speed hit probably makes that unacceptable, although it would make it easier, I suspect, to use that abstractionas part of making something which ccache or distcc could use to cache (or distribute to a cluster) all of the steps needed to build each object file. Maybe a helper C program instead?

- Ted

#5 lore raw
In reply to Theodore Tso
Quoted 7 lines
Quoted 1 line
Quoted 1 line

Theodore Tso <[email protected]> 06.01.09 18:33 >>>

Quoted 5 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line

"Theodore Ts'o" <[email protected]> 06.01.09 16:15 >>>

In the short term, though, it would be nice if we could get back a simple way of making a kernel object file using just cc, so that ccache
and distcc could be functional again. Does that seem reasonable?

Quoted 8 lines
Quoted 3 lines

Making the new logic dependent on a config option would seem reasonable to me - of course at the expense of the respective Makefile becoming even less readable.

Too late. :-) It's pretty unreadable already.... as a result, I'm not at all confident that I could make such a patch. Is this something you could perhaps whip up? I'd really appreicate it, as it would seriously speed up by kernel development efforts.

Yes, I think I could (and in fact I already put it on my to-do list), but I can't give a good prediction on when I'd be able to get to it.

Jan

#6 lore raw
Quoted 11 lines · Jan Beulich
Quoted 11 lines
Quoted 1 line
Quoted 1 line

Theodore Tso <[email protected]> 06.01.09 18:33 >>>

Quoted 8 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line

"Theodore Ts'o" <[email protected]> 06.01.09 16:15 >>>

In the short term, though, it would be nice if we could get back a simple way of making a kernel object file using just cc, so that ccache
and distcc could be functional again. Does that seem reasonable?

Making the new logic dependent on a config option would seem reasonable to me - of course at the expense of the respective Makefile becoming even less readable.

Quoted 7 lines
Quoted 4 lines

Too late. :-) It's pretty unreadable already.... as a result, I'm not at all confident that I could make such a patch. Is this something you could perhaps whip up? I'd really appreicate it, as it would seriously speed up by kernel development efforts.

Yes, I think I could (and in fact I already put it on my to-do list), but I can't give a good prediction on when I'd be able to get to it.

We only see the ccache/distcc issue if we have MODVERSIONS enabled.
So if we introduce a CONFIG option to disable strip of modules then we will have the double amount of configuration possibilities.

Today we have with or without MODVERSION.
If we make the stripping configurable then we will have in addition two different configurations.

The reason to do the .c -> .s -> .o step is to make the __crc_ symbols local so we can strip them off.

What is the gain/pain ratio here?
Would it be a possibility to drop stripping off the crc symbols and go back to the ld method that is more ccache friendly?

We would still benefir from all the other stripping done - no?

Sam

#7 lore raw
Quoted 2 lines
Quoted 1 line
Quoted 1 line

Sam Ravnborg <[email protected]> 07.01.09 12:31 >>>

What is the gain/pain ratio here?

Certainly depends on the pov - it reduces the kernel module (disk) image sizes quite a bit, so from a distro perspective its a move against the ever growing package sizes (and the disk space) hundreds or even thousands of modules require.

Quoted 3 lines

Would it be a possibility to drop stripping off the crc symbols and go back to the ld method that is more ccache friendly?

That's certainly a possibility, but I have to admit that I'd like to keep the option - a respective config option could default to 'no' if the build issues are considered more significant. I'm really not that much worried about adding another config parameter here...

Quoted 1 line

We would still benefir from all the other stripping done - no?

Yes.

Jan

#8 lore raw
Quoted 7 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line

Sam Ravnborg <[email protected]> 07.01.09 12:31 >>>

What is the gain/pain ratio here?

Certainly depends on the pov - it reduces the kernel module (disk) image sizes quite a bit, so from a distro perspective its a move against the ever growing package sizes (and the disk space) hundreds or even thousands of modules require.

The kernel modules seldom export anything so the benefit is minimal here.

I really think we are on a wrong track here.

I did a 15 minutes hack to try ripping the .c -> .s -> .o stuff out.

scripts/Makefile.build +15 -2
index 5d90030..8c93dc3 100644
@@ -153,7 +153,7 @@ $(obj)/%.i: $(src)/%.c FORCE
 
 cmd_genksyms =                                                              \
     $(CPP) -D__GENKSYMS__ $(c_flags) $< |                                   \
-    $(GENKSYMS) -T $@ -A -a $(ARCH)                                         \
+    $(GENKSYMS) -T $@ $(if $(CONFIG_KBUILD_STRIP_CRC), -A) -a $(ARCH)       \
      $(if $(KBUILD_PRESERVE),-p)                                            \
      $(if $(1),-r $(firstword $(wildcard $(@:.symtypes=.symref) /dev/null)))
 
@@ -192,6 +192,7 @@ v_file = $(@D)/.tmp_$(@F:.o=.v)
 tmp_o_file = $(@D)/.tmp_$(@F)
 no_g_c_flags = $(filter-out -g%,$(c_flags))
 
+ifdef CONFIG_KBUILD_STRIP_CRC
 cmd_cc_o_c = $(CC) $(c_flags) -S -o $(s_file) $<
 
 cmd_modversions =							\
@@ -208,8 +209,20 @@ cmd_modversions =							\
 		fi;							\
 	else								\
 		rm -f $(v_file);					\
-		$(CC) $(no_g_c_flags) -c -o $@ $(s_file);		\
+		$(CC) $(no_g_c_flags) -c -o $@ $< ;			\
 	fi;
+else
+cmd_cc_o_c = $(CC) $(c_flags) -c -o $(tmp_o_file) $<
+
+cmd_modversions =							\
+	if $(OBJDUMP) -h $(tmp_o_file) | grep -q __ksymtab; then	\
+		$(call cmd_genksyms, $(KBUILD_SYMTYPES)) > $(v_file);	\
+		$(LD) $(LDFLAGS) -r -o $@ $(tmp_o_file)	-T $(v_file);	\
+	else								\
+		rm -f $(v_file);					\
+	fi;
+
+endif
 endif
 
 ifdef CONFIG_FTRACE_MCOUNT_RECORD

Jan - I need some hard numbers to convince me that stripping the __crc symbols is worth it.

We can make it a configuration option as I showed above but we should at least know what we are gaining for this.

I am very positive about stripping of the modules in general,
but the __crc_ symbols seems to trigger a few things:
1) ccahce, distcc issues in the MODVERSION case (that is allmodconfig builds)
2) It does not increase readability of the Makefile machinery and I like more people to be able to hack on this - not less.

Thanks,
Sam

#9 lore raw
Quoted 11 lines
Quoted 1 line
Quoted 1 line

Sam Ravnborg <[email protected]> 07.01.09 14:23 >>>

Quoted 7 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line

Sam Ravnborg <[email protected]> 07.01.09 12:31 >>>

What is the gain/pain ratio here?

Certainly depends on the pov - it reduces the kernel module (disk) image sizes quite a bit, so from a distro perspective its a move against the ever growing package sizes (and the disk space) hundreds or even thousands of modules require.

The kernel modules seldom export anything so the benefit is minimal here.

Not really, for highly modular distro kernels.

Quoted 1 line

I did a 15 minutes hack to try ripping the .c -> .s -> .o stuff out.

Looks good to me. In case you decide to revert the stuff for .29, I'll merge this into my patch.

Quoted 2 lines

Jan - I need some hard numbers to convince me that stripping the __crc symbols is worth it.

The on-disk size of the modules in my builds is reduced by about 5% with this patch (but I can't immediately tell how much of this is due to __crc_* vs. other symbol stripping).

Jan

#10 lore raw
Quoted 10 lines · Jan Beulich
Quoted 10 lines
Quoted 1 line
Quoted 1 line

Sam Ravnborg <[email protected]> 07.01.09 14:23 >>>

Quoted 7 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line

Sam Ravnborg <[email protected]> 07.01.09 12:31 >>>

What is the gain/pain ratio here?

Certainly depends on the pov - it reduces the kernel module (disk) image sizes quite a bit, so from a distro perspective its a move against the ever growing package sizes (and the disk space) hundreds or even thousands of modules require.

Quoted 8 lines
Quoted 1 line

The kernel modules seldom export anything so the benefit is minimal here.

Not really, for highly modular distro kernels.

Quoted 1 line

I did a 15 minutes hack to try ripping the .c -> .s -> .o stuff out.

Looks good to me. In case you decide to revert the stuff for .29, I'll merge this into my patch.

As it stand now it:
1) breaks sparc32 build
2) causes regression with distcc/ccache use when MODVERSIONS are enabled
3) the concept is under discussion

So I will have to revert it somehow.
I will look into during a partial revert though as your patches had some stuff in them I like to keep or which does not harm (genksyms change for instance).

Quoted 7 lines
Quoted 3 lines

Jan - I need some hard numbers to convince me that stripping the __crc symbols is worth it.

The on-disk size of the modules in my builds is reduced by about 5% with this patch (but I can't immediately tell how much of this is due to __crc_* vs. other symbol stripping).

But if only on-disk size is your primary concern then another patch of Jan Engelhardt would be a bigger win:

build: install modules compressed
---
scripts/Makefile.modinst | 2 +-

1 files changed, 1 insertions(+), 1 deletions(-)
scripts/Makefile.modinst +0 -0
index efa5d94..c3421a1 100644
@@ -17,7 +17,7 @@ __modinst: $(modules)
 @:
 

quiet_cmd_modules_install = INSTALL $@

-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) \
			     $(2)/$(notdir $@) 
+      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; \
		$(mod_strip_cmd) $(2)/$(notdir $@); gzip -9f $(2)/$(notdir $@)

# Modules built outside the kernel source tree go into extra by default INSTALL_MOD_DIR ?= extra

(cut'n'paste so will not apply direct)

It does not decrease the size of vmlinux but the > 1000 modules are compressed.

The patch will in some form be present in kbuild-next.git when I open up for that.

Sam

#11 lore raw
Quoted 7 lines · Sam Ravnborg
Quoted 2 lines

Looks good to me. In case you decide to revert the stuff for .29, I'll merge this into my patch.

As it stand now it:
1) breaks sparc32 build
2) causes regression with distcc/ccache use when MODVERSIONS are enabled
3) the concept is under discussion

4) Bloats distro modules from 48M to >1GB when stripping fails. :\

regards, Kyle

#12 lore raw
In reply to Sam Ravnborg

Selon Sam Ravnborg <[email protected]>:

...

Quoted 3 lines

But if only on-disk size is your primary concern then another patch of Jan Engelhardt would be a bigger win:

Would be better to add -n to gzip flags for 2 reasons
- compressed module is smaller without file name and date include
- it make paranoid people happier. At each compilation of the same source within same environnement, using gzip without timestamp produce exactly the same binary. With timestamp include, you are no more able to track a change to the compressed module file with md5|sha1...

Even gzip-1.2.4 from 1993 support -n flag

build: install modules compressed
---
scripts/Makefile.modinst | 2 +-

1 files changed, 1 insertions(+), 1 deletions(-)
scripts/Makefile.modinst +0 -0
index efa5d94..c3421a1 100644
@@ -17,7 +17,7 @@ __modinst: $(modules)
 @:
 

quiet_cmd_modules_install = INSTALL $@

-      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; $(mod_strip_cmd) \
			     $(2)/$(notdir $@)
+      cmd_modules_install = mkdir -p $(2); cp $@ $(2) ; \
		$(mod_strip_cmd) $(2)/$(notdir $@); gzip -9nf $(2)/$(notdir $@)

# Modules built outside the kernel source tree go into extra by default INSTALL_MOD_DIR ?= extra

(cut'n'paste so will not apply direct)

Gilles

#13 lore raw
In reply to Jan Beulich
Quoted 8 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line

"Theodore Ts'o" <[email protected]> 06.01.09 16:15 >>>

In the short term, though, it would be nice if we could get back a simple way of making a kernel object file using just cc, so that ccache
and distcc could be functional again. Does that seem reasonable?

Making the new logic dependent on a config option would seem reasonable to me - of course at the expense of the respective Makefile becoming even less readable.

We need to do this I think - sigh..

The Makefile.build bits should be easiest.
Maybe I should revisit the patch to move vmlinux link to a shell script to make the toplevel Makefile simpler??

Sam

#14 lore raw
In reply to Theodore Ts'o

From: "Theodore Ts'o" <[email protected]>
Date: Tue, 06 Jan 2009 10:15:26 -0500

Quoted 8 lines

The following commit, ad7a953c, "kbuild: strip generated symbols from *.ko" which was merged to the mainline last week (December 28th) has the unfortunate side effct of making ccache useless. That's because all files are now built using a two step process. First they are compiled to assembly via "gcc -S -o .tmp_foo.s" and then assembled via "gcc" in a separate step. Unfortunately, ccache doesn't seem to be able to cache either the compilation or the assembly step (and it may not be worth it once the two steps are separated).

I think the problem might be that the assembler source always "changes" because a different date or timestamp string is put in there by gcc every time the assembler source is built.

#15 lore raw
Quoted 6 lines · David Miller

From: "Theodore Ts'o" <[email protected]>
Date: Tue, 06 Jan 2009 10:15:26 -0500

I think the problem might be that the assembler source always "changes" because a different date or timestamp string is put in there by gcc every time the assembler source is built.

I'm not sure that's the problem, actually. Running "ccache -s" doesn't show any additional cache misses *or* hits. Instead the only statistic which gets bumped is the "called for link" counter. So I suspect it's because ccache doesn't attempt to cache "gcc -S" or "gcc -c foo.s" operations. So it might be the case that a change to ccache to cache compile-to-assemble and assemble-to-object code transformation would help, but then we would still end up calling ccache twice, and double the number of things that would be cached in the ccache directories, not to mention doubling the number of calls to remote hosts of distcc is involved.

So a full, long-term solution that gets back all of the efficiency wins of ccache and distcc probably requires somehow passing a script or program to ccache and distcc which does all of the steps which kbuild needs to do to transform a kernel .c file to a kernel .o file, which seems to involve interposing multiple steps between the foo.c to foo.s and foo.s and foo.o steps, as well as post-processing on the resulting .o file using objcopy. If we can somehow get ccache to cache all of that work, and distribute all of the steps of going from foo.c to foo.o to other machines when distcc is involved, that would be really cool. I suspect this is not trivial to do, however....

- Ted

#16 lore raw
In reply to Theodore Ts'o
Quoted 8 lines · Theodore Ts'o

The following commit, ad7a953c, "kbuild: strip generated symbols from *.ko" which was merged to the mainline last week (December 28th) has the unfortunate side effct of making ccache useless. That's because all files are now built using a two step process. First they are compiled to assembly via "gcc -S -o .tmp_foo.s" and then assembled via "gcc" in a separate step. Unfortunately, ccache doesn't seem to be able to cache either the compilation or the assembly step (and it may not be worth it once the two steps are separated).

Hi Ted.

How about this simple patch.

scripts/Makefile.build +1 -1
index 5d90030..3b9b8f5 100644
@@ -208,7 +208,7 @@ cmd_modversions =							\
 		fi;							\
 	else								\
 		rm -f $(v_file);					\
-		$(CC) $(no_g_c_flags) -c -o $@ $(s_file);		\
+		$(CC) $(no_g_c_flags) -c -o $@ $< ;			\
 	fi;
 endif
 

It basically starts all over again with the C file if the file did not export any symbols (typical case).

This I think should allow ccache to be functional again.

I do not propose this as a final suggestion - more so that I'm sure I understand the problem.

Sam

#17 lore raw
Quoted 6 lines · Sam Ravnborg

Hi Ted.

How about this simple patch.

It basically starts all over again with the C file if the file did not export any symbols (typical case).

I tried the patch. It does cause ccache to be functional again, at least as far as causing the "cache hit" stats to get bumped. However, the ccache "called for link" stats is still going up (which is I guess how ccache misleadingly categorizes "cc -S" requests to compile to assembly). However, it doesn't actually save time; quite the reverse, it makes things worse. Doing a 32-bit compile, I tried compiling a kernel where after each compile, I remove the ext4 object files via "/bin/rm ../ext4-32/fs/ext4/*.o". Without the patch, these were the results after getting the cache warm, running the compile three tims, and taking the average:

real 36.238 // user 44.143 // sys 11.375

Applying the patch, it actually took longer to recompile fs/ext4/*.o and relink vmlinuz: (again average of three compiles)

real 38.401 // user 46.438 // sys 11.838

That's because ccache has to do the ccache -E and then calculate the checksum to look up the results of the cache. Before the patch, we are doing this:

preprocessor, compile, check for exported sysmbols, assemble

after the patch, we are doing this:

preprocessor, compile, check for exported symbols, then (in ccache), preprocessor, locate the cached object file, write cached objectfile.

What this experiment shows is that even with a completely warm cache, it's faster to run the assembler than to run ccache and have it re-run the preprocessor and look up the cached object file. In fact, it's quite a bit faster.

So basically, in order to make things work well with ccache, we really need to avoid redoing the compile unconditionally. If the goal is just to check to see if there were any exported symbols, we would be better off doing a grep for EXPORT_SYMBOL.

In contrast, with 2.6.28, the numbers for just compiling fs/ext4/*.o and relinking vmlinuz is.

real 28.836 // user 30.949 // sys 10.562

Note that the bulk of this time is doing the kbuild setup and relinking vmlinuz. The amount of time to do setup and relink is 18.677 / 21.639 / 8.253. So if you subtract that out, and simply compare the time to compile fs/ext4/*.o, what you get is:

2.6.28: real 10.159 user 9.310 sys 2.309
2.6.28-git7: real 17.561 user 22.503 sys 3.122
2.6.28-git7 w/your patch real 19.724 user 24.799 sys 3.585

Bottom line, I suspect if we want to make builds fast, we really need to cache the full kbuild pipeline. I hope you find these measurements useful.

- Ted

#18 lore raw
In reply to Theodore Ts'o
Quoted 7 lines · Theodore Ts'o

Or, if that's too complicated, maybe it would be worthwhile to have kbuild create its own specialized ccache system? Note that the last two solutions rule out using distcc, unless we can encapsulate the build process from a series of Makefile macros to a shell or C program, which could then be injected to the remote host system to be executed by distcc. One value of doing that is the CRC or MD5 of the shell script could be used as the version tag for the cache system.

Ho-hum... Could somebody explain why the hell had we switched to this "intermediate .s" approach, anyway? It's not as if we couldn't run objcopy after what we used to do...

BTW, do we care about non-ELF targets at all?

#19 lore raw
Quoted 10 lines
Quoted 1 line
Quoted 1 line

Al Viro <[email protected]> 07.01.09 06:10 >>>

Quoted 7 lines · Theodore Ts'o

Or, if that's too complicated, maybe it would be worthwhile to have kbuild create its own specialized ccache system? Note that the last two solutions rule out using distcc, unless we can encapsulate the build process from a series of Makefile macros to a shell or C program, which could then be injected to the remote host system to be executed by distcc. One value of doing that is the CRC or MD5 of the shell script could be used as the version tag for the cache system.

Quoted 3 lines

Ho-hum... Could somebody explain why the hell had we switched to this "intermediate .s" approach, anyway? It's not as if we couldn't run objcopy after what we used to do...

Because objcopy wouldn't remove the __crc_* symbols the way the object files were generated previously. I explored all possibilities I could think of, with that two step process being the last, but the only one that had the intended effect. In the end I queried the binutils list, and got confirmation that there's no (existing) way to get rid of symbols used in relocations in an already existing object file without corrupting it.

Jan

#20 lore raw
Quoted 14 lines · Jan Beulich
Quoted 10 lines
Quoted 1 line
Quoted 1 line

Al Viro <[email protected]> 07.01.09 06:10 >>>

Quoted 7 lines · Theodore Ts'o

Or, if that's too complicated, maybe it would be worthwhile to have kbuild create its own specialized ccache system? Note that the last two solutions rule out using distcc, unless we can encapsulate the build process from a series of Makefile macros to a shell or C program, which could then be injected to the remote host system to be executed by distcc. One value of doing that is the CRC or MD5 of the shell script could be used as the version tag for the cache system.

Quoted 3 lines

Ho-hum... Could somebody explain why the hell had we switched to this "intermediate .s" approach, anyway? It's not as if we couldn't run objcopy after what we used to do...

Quoted 6 lines

Because objcopy wouldn't remove the __crc_* symbols the way the object files were generated previously. I explored all possibilities I could think of, with that two step process being the last, but the only one that had the intended effect. In the end I queried the binutils list, and got confirmation that there's no (existing) way to get rid of symbols used in relocations in an already existing object file without corrupting it.

I do not understand two things.
a) WTF would we care about ones in vmlinux, when kallsyms drops the
absolute ones anyway?
b) is there any reason why we can't copy symbol table out of module
ourselves (instead of setting SHF_ALLOC on it) and trim the crap out of it? Note that __crc_... is not the only junk in there - you don't need (or want) to keep the things like undefs for /proc/kallsyms purposes. And modules *using* an exported symbol are more common that modules exporting one... Why not copy it at add_kallsyms() time, skipping the junk we don't want anyway?

#21 lore raw
Quoted 4 lines
Quoted 1 line
Quoted 1 line

Al Viro <[email protected]> 07.01.09 15:03 >>>

I do not understand two things.
a) WTF would we care about ones in vmlinux, when kallsyms drops the
absolute ones anyway?

The patch is not only about __crc_* - for vmlinux, this part really doesn't matter, but there's a significant amount of other symbols that get stripped from vmlinux with the second patch.

Quoted 7 lines

b) is there any reason why we can't copy symbol table out of module
ourselves (instead of setting SHF_ALLOC on it) and trim the crap out of it? Note that __crc_... is not the only junk in there - you don't need (or want) to keep the things like undefs for /proc/kallsyms purposes. And modules *using* an exported symbol are more common that modules exporting one... Why not copy it at add_kallsyms() time, skipping the junk we don't want anyway?

Would certainly be doable, but would seem reasonable only if it's a small set of easily recognizable symbols. If you look at scripts/strip-symbols, you may agree that it's better to do this at build time...

Jan

#22 lore raw
Quoted 11 lines · Jan Beulich
Quoted 7 lines

b) is there any reason why we can't copy symbol table out of module
ourselves (instead of setting SHF_ALLOC on it) and trim the crap out of it? Note that __crc_... is not the only junk in there - you don't need (or want) to keep the things like undefs for /proc/kallsyms purposes. And modules *using* an exported symbol are more common that modules exporting one... Why not copy it at add_kallsyms() time, skipping the junk we don't want anyway?

Would certainly be doable, but would seem reasonable only if it's a small set of easily recognizable symbols. If you look at scripts/strip-symbols, you may agree that it's better to do this at build time...

I will not. Undefined ones are far more common than any of those...

#23 lore raw
Quoted 10 lines
Quoted 1 line
Quoted 1 line

Al Viro <[email protected]> 07.01.09 15:37 >>>

Quoted 8 lines · Jan Beulich
Quoted 7 lines

b) is there any reason why we can't copy symbol table out of module
ourselves (instead of setting SHF_ALLOC on it) and trim the crap out of it? Note that __crc_... is not the only junk in there - you don't need (or want) to keep the things like undefs for /proc/kallsyms purposes. And modules *using* an exported symbol are more common that modules exporting one... Why not copy it at add_kallsyms() time, skipping the junk we don't want anyway?

Quoted 5 lines
Quoted 3 lines

Would certainly be doable, but would seem reasonable only if it's a small set of easily recognizable symbols. If you look at scripts/strip-symbols, you may agree that it's better to do this at build time...

I will not. Undefined ones are far more common than any of those...

Undefined ones could certainly be stripped out there (while they definitely can't be at build time).

Jan

#24 lore raw
In reply to Theodore Ts'o
Quoted 3 lines · Theodore Ts'o

The following commit, ad7a953c, "kbuild: strip generated symbols from *.ko" which was merged to the mainline last week (December 28th) has the unfortunate side effct of making ccache useless.

I've just discovered another problem with this commit.

% du -s /lib/modules/2.6.28
63072 /lib/modules/2.6.28/
du -s /lib/modules/2.6.28-git7
221556 /lib/modules/2.6.28-git7/

The two configs are largely equal. I'm building with CONFIG_DEBUG_KERNEL=y, but then installing with "make INSTALL_MOD_STRIP=1 modules_install".

But look:

% ls /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko 10460 /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko

% ls /lib/modules/2.6.28/kernel/fs/xfs/xfs.ko
564 /lib/modules/2.6.28/kernel/fs/xfs/xfs.ko

The reason? The ad7a953c commit uses this procedure to do the installation:

# objcopy --strip-debug --wildcard --strip-symbols /usr/projects/linux/linux-2.6/scripts/strip-symbols fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko  || cp fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko 
objcopy: not stripping symbol `__ksymtab_strings' because it is named in a relocation

So because it can't safely strip __ksymtab_strings it returns a non-zero exit status, and doesn't do anything at all, and then the cp command doesn't do any stripping whatso ever.

I could fix this by patching Makefile and scripts/Makefile.modist to do this instead:

# objcopy --strip-debug --wildcard --strip-symbols /usr/projects/linux/linux-2.6/scripts/strip-symbols fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko  || (cp fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko ; strip --strip-debug /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko)

... but at this point, is there going to be any massive downside if I just revert commit ad7a953c? Quite frankly, it's causing me a huge amount of trouble, and I'm still a bit unclear what the upside of this patch is. As near as I can tell there is *single* __crc_ symbol in xfs.ko which all of this rigamorale is doing to strip out. From what I can tell, not quite doubling the compile time when fully cached by ccache, causing INSTALL_MOD_STRIP to fail randomly so that the installed modules are a factor of 4 larger, compromising the amount of space in my root partition, is all to remove a handful of __crc_* symbols from the generated .ko file?

What am I missing? Why is stripping the __crc_* symbols so gosh-darned important?

- Ted

#25 lore raw
Quoted 2 lines
Quoted 1 line
Quoted 1 line

Theodore Tso <[email protected]> 07.01.09 14:12 >>>

objcopy: not stripping symbol `__ksymtab_strings' because it is named in a relocation

Another thing I've never seen in any of my builds - these strings always get relocations against the corresponding section symbol.

Quoted 8 lines

So because it can't safely strip __ksymtab_strings it returns a non-zero exit status, and doesn't do anything at all, and then the cp command doesn't do any stripping whatso ever.

I could fix this by patching Makefile and scripts/Makefile.modist to do this instead:

# objcopy --strip-debug --wildcard --strip-symbols /usr/projects/linux/linux-2.6/scripts/strip-symbols fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko  || (cp fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko ; strip --strip-debug /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko)

I'd rather insert a second objcopy attempt here with just --strip-debug.

Quoted 10 lines

... but at this point, is there going to be any massive downside if I just revert commit ad7a953c? Quite frankly, it's causing me a huge amount of trouble, and I'm still a bit unclear what the upside of this patch is. As near as I can tell there is *single* __crc_ symbol in xfs.ko which all of this rigamorale is doing to strip out. From what I can tell, not quite doubling the compile time when fully cached by ccache, causing INSTALL_MOD_STRIP to fail randomly so that the installed modules are a factor of 4 larger, compromising the amount of space in my root partition, is all to remove a handful of __crc_* symbols from the generated .ko file?

With this and the previous issue I certainly wouldn't mind this and the subsequent vmlinux stripping patches to be reverted, and I'd aim at providing an improved version for .30.

Quoted 2 lines

What am I missing? Why is stripping the __crc_* symbols so gosh-darned important?

In a separate response I already explained what the goal here is - size reduction (i.e. making distro package sizes as well as foot print on disk smaller, thus improving the already existing stripping). And it's not only about __crc_*, those are just the ones that when removed have the biggest overall effect. And, as pointed out above, the vmlinux stripping patch, which has an even more significant effect as it reduces the in- memory kernel size when using KALLSYMS_ALL, depends on this one.

I have to admit that I generally dislike the tendency for everything to always only grow, and hence I'm trying to find opportunities to shrink stuff where possible. I've got a couple of other patches in my local queue that also aim in that direction, plus there are a few thoughts on improvements beyond that.

Jan

#26 lore raw
Quoted 5 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line

Theodore Tso <[email protected]> 07.01.09 14:12 >>>

objcopy: not stripping symbol `__ksymtab_strings' because it is named in a relocation

Another thing I've never seen in any of my builds - these strings always get relocations against the corresponding section symbol.

Just to be clear; it doesn't always fail; it seems to be random (or at least, I haven't had time to try to suss out why sometimes __ksymtab_strings is being used in a relocation):

INSTALL fs/cifs/cifs.ko
INSTALL fs/coda/coda.ko
INSTALL fs/configfs/configfs.ko
objcopy: not stripping symbol `__ksymtab_strings' because it is named in a relocation
INSTALL fs/dlm/dlm.ko
objcopy: not stripping symbol `__ksymtab_strings' because it is named in a relocation
INSTALL fs/ecryptfs/ecryptfs.ko
INSTALL fs/efs/efs.ko
INSTALL fs/ext2/ext2.ko
INSTALL fs/fat/fat.ko
objcopy: not stripping symbol `__ksymtab_strings' because it is named in a relocation
INSTALL fs/fat/msdos.ko
INSTALL fs/fat/vfat.ko
INSTALL fs/freevxfs/freevxfs.ko

This is on an Ubuntu Hardy system, if this helps. I'm using gcc 4.2.4, and binutils version 2.18.1~cvs20080103-0ubuntu1.

Quoted 3 lines
Quoted 1 line
# objcopy --strip-debug --wildcard --strip-symbols /usr/projects/linux/linux-2.6/scripts/strip-symbols fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko  || (cp fs/xfs/xfs.ko /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko ; strip --strip-debug /lib/modules/2.6.28-git7/kernel/fs/xfs/xfs.ko)

I'd rather insert a second objcopy attempt here with just --strip-debug.

Yes, that makes sense, of course.

Quoted 7 lines

In a separate response I already explained what the goal here is - size reduction (i.e. making distro package sizes as well as foot print on disk smaller, thus improving the already existing stripping). And it's not only about __crc_*, those are just the ones that when removed have the biggest overall effect. And, as pointed out above, the vmlinux stripping patch, which has an even more significant effect as it reduces the in- memory kernel size when using KALLSYMS_ALL, depends on this one.

I'm all for size reduction; the problem is that I don't have one of Linus's fancy machines that can do 10 second full kernel compiles. So I'm very much reliant on ccache, and a 70% increase in kernel compilation time, when a full compile already takes order of magnitude 5-10 minutes (this is for my personal no-modules, stripped-down development build), is very painful and easily noticed.

Maybe we need to modify ccache, or pull ccache into the kernel sources and then hack it for our unique, and apparently , constantly growing-in-complexity build complexity. One thing that would really help is some documentation and some way of seeing *all* of the steps that are getting executed when we compile a .c to .o file. One of the things I noticed is that make V=1 only showed the cc -S step, but it seemd to mask out the other steps of what was going on, which forced me to go diving into the kbuild scripts --- and for someone who isn't daily hacking the kbuild scripts, and who needed to trace all of the macros, and who might not know at their fingertips what $(filter-out ...), $(wildcard ...), and $(notdir ...) do --- it's highly painful.

I'm sure this is just my own ignorance, but a tutorial in how the kbuild system works, which files are involved, where to look for various things, and a short summary to all of the gmake magic which gets used would be very helpful.

Quoted 5 lines

I have to admit that I generally dislike the tendency for everything to always only grow, and hence I'm trying to find opportunities to shrink stuff where possible. I've got a couple of other patches in my local queue that also aim in that direction, plus there are a few thoughts on improvements beyond that.

I'm certainly sympathetic to this. I just also have this desire to make sure that it is possible to do kernel hacking on a X61-class laptop, and that mongo-expensive 8 socket, quad-core systems with vast amounts of memories aren't necessary in order to do compile-edit-debug cycles in reasonable amounts of time. :-/

(And BTW, I consider the pre-2.6.28 compile times only barely acceptable; most of this isn't kbuild's fault, I know, it includes the fact that link steps are still slow, and gcc is constantly getting increasingly slower over time. Sigh... I will just note wryly that it used to be that I could compile 0.9x kernels on a 40 MHz 386 machine in 10 minutes. Some 15 years later, it still takes roughly the same amount of time to compile a kernel, even though computers have gotten vastly faster since then. Something seems wrong with that.... :-)

- Ted

#27 lore raw
Quoted 11 lines
Quoted 1 line
Quoted 1 line

Theodore Tso <[email protected]> 07.01.09 15:28 >>>

Quoted 5 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line

Theodore Tso <[email protected]> 07.01.09 14:12 >>>

objcopy: not stripping symbol `__ksymtab_strings' because it is named in a relocation

Another thing I've never seen in any of my builds - these strings always get relocations against the corresponding section symbol.

Just to be clear; it doesn't always fail; it seems to be random (or at least, I haven't had time to try to suss out why sometimes __ksymtab_strings is being used in a relocation):

It's apparently happening only for modules which export symbols.

Jan

#28 lore raw
In reply to Theodore Tso
Quoted 7 lines · Theodore Tso
Quoted 3 lines · Theodore Ts'o

The following commit, ad7a953c, "kbuild: strip generated symbols from *.ko" which was merged to the mainline last week (December 28th) has the unfortunate side effct of making ccache useless.

I've just discovered another problem with this commit.

Quoted 8 lines

% du -s /lib/modules/2.6.28
63072 /lib/modules/2.6.28/
du -s /lib/modules/2.6.28-git7
221556 /lib/modules/2.6.28-git7/

The two configs are largely equal. I'm building with CONFIG_DEBUG_KERNEL=y, but then installing with "make INSTALL_MOD_STRIP=1 modules_install".

Fedora rawhide users noticed this same problem this morning too, when I pushed out a build that eats up a gig of diskspace thanks to the number of modules we have, that are all afflicted by this problem.

I'll have to revert ad7a953c for our builds to continue tracking mainline without monopolising the diskspace in the Fedora build system.

Dave

Signature · 1 line
http://www.codemonkey.org.uk
#29 lore raw
Quoted 7 lines · Dave Jones

Fedora rawhide users noticed this same problem this morning too, when I pushed out a build that eats up a gig of diskspace thanks to the number of modules we have, that are all afflicted by this problem.

I'll have to revert ad7a953c for our builds to continue tracking mainline without monopolising the diskspace in the Fedora build system.

Note: you'll probably also have revert 9bb48247 (allow stripping of generated symbols under CONFIG_KALLSYMS_ALL) since I believe it depends on ad7a953c.

- Ted

#30 lore raw
Quoted 6 lines · Theodore Tso
Quoted 5 lines · Dave Jones

Fedora rawhide users noticed this same problem this morning too, when I pushed out a build that eats up a gig of diskspace thanks to the number of modules we have, that are all afflicted by this problem.

Quoted 7 lines
Quoted 2 lines

I'll have to revert ad7a953c for our builds to continue tracking mainline without monopolising the diskspace in the Fedora build system.

Note: you'll probably also have revert 9bb48247 (allow stripping of generated symbols under CONFIG_KALLSYMS_ALL) since I believe it depends on ad7a953c.

This doesn't appear to be the culprit. Or at least, doesn't fix the problem.

make INSTALL_MOD_PATH=mods/ modules_install

with a distro kernel config (ie: Fedora's[1]) still results in 1.8GB of modules (willy checked that it wasn't a Fedora toolchain bug by reproducing on a Debian system.)

As near as I can ascertain, this is limited to x86_64, at least on Fedora:

kyle@localhost ~/misc $ for i in *.rpm; do echo $i; rpm
-qip $i | grep Size; done
kernel-2.6.29-0.25.rc0.git14.fc11.ppc64.rpm
Size : 124858478 License:
kernel-2.6.29-0.25.rc0.git14.fc11.ppc.rpm
Size : 66313693 License:
kernel-2.6.29-0.25.rc0.git14.fc11.x86_64.rpm
Size : 1041480317 License:
kernel-PAE-2.6.29-0.25.rc0.git14.fc11.i686.rpm
Size : 56252360 License:

x86_64 is an order of magnitude bigger than the rest, which have identical debug options enabled.

regards, Kyle

#31 lore raw
Quoted 10 lines · Kyle McMartin
Quoted 6 lines
Quoted 2 lines

I'll have to revert ad7a953c for our builds to continue tracking mainline without monopolising the diskspace in the Fedora build system.

Note: you'll probably also have revert 9bb48247 (allow stripping of generated symbols under CONFIG_KALLSYMS_ALL) since I believe it depends on ad7a953c.

This doesn't appear to be the culprit. Or at least, doesn't fix the problem.

Quoted 8 lines

make INSTALL_MOD_PATH=mods/ modules_install

with a distro kernel config (ie: Fedora's[1]) still results in 1.8GB of modules (willy checked that it wasn't a Fedora toolchain bug by reproducing on a Debian system.)

As near as I can ascertain, this is limited to x86_64, at least on Fedora:

It shows up for me with x86_32, and it is fixed by reverting those two commits. Pre-2.6.28-rc6:

74864 -rw-r--r-- 1 tytso tytso 76657780 2008-12-12 14:41 linux-image-2.6.28-rc8-00066-gc810663_tytso.SMP.1_i386.deb

Post 2.6.28:

146380 -rw-r--r-- 1 tytso tytso 149888992 2009-01-06 02:01 linux-image-2.6.28-05784-gd5b9778_tytso.SMP.1_i386.deb

After I revert the above two commits:

76340 -rw-r--r-- 1 tytso tytso 78167848 2009-01-09 08:45 linux-image-2.6.28-07956-g2e69ab5_tytso.SMP.1_i386.deb

- Ted

#32 lore raw
In reply to Kyle McMartin
Quoted 4 lines · Kyle McMartin

with a distro kernel config (ie: Fedora's[1]) still results in 1.8GB of modules (willy checked that it wasn't a Fedora toolchain bug by reproducing on a Debian system.)

P.S. I have a hacked make-kpkg which installs the modules via:

make INSTALL_MOD_STRIP=1 modules_install

Unfortunately, I'm not using kbuild's make install builddeb since it doesn't create the headers and debuginfo files; and the debian's official kernel building scripts (make-kpkg has been deprecated) doesn't correctly deal with creating debuginfo files, either.

In any case, its make INSTALL_MOD_STRIP=1 which is getting busted by the two commits, and which also spike out ccache and distcc.

- Ted

#33 lore raw
Quoted 10 lines · Theodore Tso
Quoted 4 lines · Kyle McMartin

with a distro kernel config (ie: Fedora's[1]) still results in 1.8GB of modules (willy checked that it wasn't a Fedora toolchain bug by reproducing on a Debian system.)

P.S. I have a hacked make-kpkg which installs the modules via:

make INSTALL_MOD_STRIP=1 modules_install

Quoted 8 lines

Unfortunately, I'm not using kbuild's make install builddeb since it doesn't create the headers and debuginfo files; and the debian's official kernel building scripts (make-kpkg has been deprecated) doesn't correctly deal with creating debuginfo files, either.

In any case, its make INSTALL_MOD_STRIP=1 which is getting busted by the two commits, and which also spike out ccache and distcc.

Right, I had a braino and forgot that having 1.8GB of modules is actually expected. The broken stage is the stripping of the debuginfo into a seperate set of files, so it's likely Fedora-specific.

regards, Kyle

#34 lore raw
In reply to Theodore Ts'o
Quoted 27 lines · Theodore Ts'o

The following commit, ad7a953c, "kbuild: strip generated symbols from *.ko" which was merged to the mainline last week (December 28th) has the unfortunate side effct of making ccache useless. That's because all files are now built using a two step process. First they are compiled to assembly via "gcc -S -o .tmp_foo.s" and then assembled via "gcc" in a separate step. Unfortunately, ccache doesn't seem to be able to cache either the compilation or the assembly step (and it may not be worth it once the two steps are separated).

It would be nice if there was a CONFIG option which compiles files the old-fashioned way, even if we lose the ability to strip the__crc_ symbols, just so that ccache can function again, since it can really speed up the kernel development process. Alternatively, maybe ccache could somehow be taught a specialized way take the .c file, some kind of kbuild-specified version identifier, and then output of the gcc -S, assembly frobnication, gcc -c, objcopy pipeline that kbuild uses to build an object file, so we can have cache the entire kbuild object building process.

Or, if that's too complicated, maybe it would be worthwhile to have kbuild create its own specialized ccache system? Note that the last two solutions rule out using distcc, unless we can encapsulate the build process from a series of Makefile macros to a shell or C program, which could then be injected to the remote host system to be executed by distcc. One value of doing that is the CRC or MD5 of the shell script could be used as the version tag for the cache system.

Quoted 3 lines

In the short term, though, it would be nice if we could get back a simple way of making a kernel object file using just cc, so that ccache
and distcc could be functional again. Does that seem reasonable?

So what's the current status on this, could we get it reverted ASAP or add that CONFIG_ switch (default N)?

Leaving the build system broken for so long just isn't cool.

#35 lore raw
Quoted 5 lines
Quoted 1 line
Quoted 1 line

Peter Zijlstra <[email protected]> 14.01.09 18:16 >>>

So what's the current status on this, could we get it reverted ASAP or add that CONFIG_ switch (default N)?

Leaving the build system broken for so long just isn't cool.

It is my understanding that Sam was looking at doing a partial revert first. If that doesn't work out, doing the CONFIG_* thing would be pretty trivial (I merged it into my local patch, and hence would simply have to extract it).

Sam?

Jan

#36 lore raw
Quoted 9 lines · Jan Beulich
Quoted 5 lines
Quoted 1 line
Quoted 1 line

Peter Zijlstra <[email protected]> 14.01.09 18:16 >>>

So what's the current status on this, could we get it reverted ASAP or add that CONFIG_ switch (default N)?

Leaving the build system broken for so long just isn't cool.

It is my understanding that Sam was looking at doing a partial revert first. If that doesn't work out, doing the CONFIG_* thing would be pretty trivial (I merged it into my local patch, and hence would simply have to extract it).

I just have a local branch, "emergency-fixes" which just reverts commits ad7a953c and 9bb48247, which I pull in before doing full builds.

You can also work around the bug by disabling CONFIG_MODVERSIONS; one of my build configurations has almost no modules at all but it still had CONFIG_MODVERSIONS disabled, so for my sniff tests build of mainline where I don't want to pull in the emergency-fixes branch, I've also dealt with the situation by disabling CONFIG_MODVERSIONS.

These are both hacks, but they do work around the problem...

- Ted