build reproducibility
Hello,
If I do the following:
git clean -dfx
cp saved_config .config
make olddefconfig && make && make modules_install && make install
Should I always end up with the same kernel, regardless of the kernel that is currently running on the machine?
I see a large performance difference between Linux 5.10 and all versions afterwards for a particular benchmark. I am unable to bisect the problem eg between 5.10 and 5.11, because as soon as I come to a kernel that gives the bad performance, all of the kernels that I generate subsequently in the bisecting process (using the above commands) also have the bad performance.
It could of course be that I have completely misinterpreted the problem, and it has nothing to do with the kernel. But I have tested the program a lot when only working on variants of Linux 5.9. I only start to have problems when I use versions >= 5.11.
thanks,
julia
Quoted 18 lines · Julia Lawall
Hello,
If I do the following:
git clean -dfx
cpsaved_config.config
make olddefconfig && make && makemodules_install&& make installShould I always end up with the same kernel, regardless of the kernel that is currently running on the machine?
I see a large performance difference between Linux 5.10 and all versions afterwards for a particular benchmark. I am unable to bisect the problem eg between 5.10 and 5.11, because as soon as I come to a kernel that gives the bad performance, all of the kernels that I generate subsequently in the bisecting process (using the above commands) also have the bad performance.
Quoted 4 lines
It could of course be that I have completely misinterpreted the problem, and it has nothing to do with the kernel. But I have tested the program a lot when only working on variants of Linux 5.9. I only start to have problems when I use versions >= 5.11.
Hi,
My "guess" is that this has something to do with the build reusing some current file(s) that need to be rebuilt. I.e., adding a "make clean" or "make proper" might be needed.
I say this only because sometimes I cannot even reproduce a build that has errors or warnings unless I prefix it with make clean or mrproper. (i.e., nothing to do with booting and running the new kernel)
Even though the .config file has changed and I do "make olddefconfig", the same build errors do not show up unless I do the clean or mrproper step also.
Signature · 1 line
~Randy
Quoted 24 lines · Randy Dunlap
Quoted 18 lines · Julia Lawall
Hello,
If I do the following:
git clean -dfx
cpsaved_config.config
make olddefconfig && make && makemodules_install&& make installShould I always end up with the same kernel, regardless of the kernel that is currently running on the machine?
I see a large performance difference between Linux 5.10 and all versions afterwards for a particular benchmark. I am unable to bisect the problem eg between 5.10 and 5.11, because as soon as I come to a kernel that gives the bad performance, all of the kernels that I generate subsequently in the bisecting process (using the above commands) also have the bad performance.
Quoted 4 lines
It could of course be that I have completely misinterpreted the problem, and it has nothing to do with the kernel. But I have tested the program a lot when only working on variants of Linux 5.9. I only start to have problems when I use versions >= 5.11.
Quoted 5 lines
Hi,
My "guess" is that this has something to do with the build reusing some current file(s) that need to be rebuilt. I.e., adding a "make clean" or "make proper" might be needed.
This was my guess too. But I have the git clean -dfx. I did a comparison with make distclean and this does a little more (mostly some files in tools).
thanks,
julia
Quoted 13 lines
I say this only because sometimes I cannot even reproduce a build that has errors or warnings unless I prefix it with make clean or mrproper. (i.e., nothing to do with booting and running the new kernel)
Even though the .config file has changed and I do "make olddefconfig", the same build errors do not show up unless I do the clean or mrproper step also.--
~Randy
Quoted 35 lines · Julia Lawall
Quoted 24 lines · Randy Dunlap
Quoted 18 lines · Julia Lawall
Hello,
If I do the following:
git clean -dfx
cpsaved_config.config
make olddefconfig && make && makemodules_install&& make installShould I always end up with the same kernel, regardless of the kernel that is currently running on the machine?
I see a large performance difference between Linux 5.10 and all versions afterwards for a particular benchmark. I am unable to bisect the problem eg between 5.10 and 5.11, because as soon as I come to a kernel that gives the bad performance, all of the kernels that I generate subsequently in the bisecting process (using the above commands) also have the bad performance.
Quoted 4 lines
It could of course be that I have completely misinterpreted the problem, and it has nothing to do with the kernel. But I have tested the program a lot when only working on variants of Linux 5.9. I only start to have problems when I use versions >= 5.11.
Quoted 5 lines
Hi,
My "guess" is that this has something to do with the build reusing some current file(s) that need to be rebuilt. I.e., adding a "make clean" or "make proper" might be needed.
Quoted 7 lines
This was my guess too. But I have the git clean -dfx. I did a comparison with make distclean and this does a little more (mostly some files in tools).
thanks,
julia
'git clean -dfx' is a very hard cleaning.
So, you are doing a full build in every step of bisecting.
I have no idea to explain the symptom you observed:
"as soon as I come to a kernel that gives
the bad performance, all of the kernels that I generate subsequently in the bisecting process"
If you desire perfect reproducibility, you can check
Documentation/kbuild/reproducible-builds.rst
But, I doubt slight differences such as timestamps can explain the large performance difference.
If you are chasing the performance issue,
commit cf536e185869d4815 said
CONFIG_DEBUG_FORCE_FUNCTION_ALIGN_*
might be useful to eliminate the possibility
of code alignment.
Otherwise, I have no more idea...
Signature · 2 lines
Best Regards Masahiro Yamada
Hello Julia,
Quoted 8 lines · Julia Lawall
Quoted 3 lines · Randy Dunlap
My "guess" is that this has something to do with the build reusing some current file(s) that need to be rebuilt. I.e., adding a "make clean" or "make proper" might be needed.
This was my guess too. But I have the git clean -dfx. I did a comparison with make distclean and this does a little more (mostly some files in tools).
Have you tried power-cycling the machine between boots, or just rebooting on a working kernel before booting again on a faulty one ? It could be possible that "something" changes a hardware setting that the BIOS does not touch, leaving your machine in a different state after you've booted the first problematic kernel. For example, it's possible to set some CPU MSRs that affect the maximum CPU power, hence its performance. Normally the BIOS should reset them, but for this it must know about the one your kernel (or even userland) would set.
Hoping this helps,
Willy
Quoted 12 lines · Willy Tarreau
Hello Julia,
Quoted 8 lines · Julia Lawall
Quoted 3 lines · Randy Dunlap
My "guess" is that this has something to do with the build reusing some current file(s) that need to be rebuilt. I.e., adding a "make clean" or "make proper" might be needed.
This was my guess too. But I have the git clean -dfx. I did a comparison with make distclean and this does a little more (mostly some files in tools).
Quoted 8 lines
Have you tried power-cycling the machine between boots, or just rebooting on a working kernel before booting again on a faulty one ? It could be possible that "something" changes a hardware setting that the BIOS does not touch, leaving your machine in a different state after you've booted the first problematic kernel. For example, it's possible to set some CPU MSRs that affect the maximum CPU power, hence its performance. Normally the BIOS should reset them, but for this it must know about the one your kernel (or even userland) would set.
OK, thanks for the suggestions. My impression is that there is a real performance problem in 5.11. The part I don't understand is why once I have booted that kernel, all of the kernels I make afterwards have the same performance characteristics.
If I do git clean -dfx, then copy a fixed configuration to .config, and then use make olddefconfig, should anything about the currently running kernel have an impact on the kernel that is produced?
I'll try simply rebooting the machine on each git bisect step. That should eliminate one more aspect of local state.
thanks,
julia
Quoted 8 lines · Julia Lawall
My impression is that there is a real
performance problem in 5.11. The part I don't understand is why once I have booted that kernel, all of the kernels I make afterwards have the same performance characteristics.If I do git clean -dfx, then copy a fixed configuration to .config, and then use make olddefconfig, should anything about the currently running kernel have an impact on the kernel that is produced?
Normally not at all, especially if you restart from a fixed .config. By the way, you should compare the resulting .config after "make oldconfig" for all your kernels, in case you spot a difference there, but there is no reason for that difference to depend on the currently running kernel. Or maybe it detects something related to your machine and adjusts the .config accordingly, and that detection depends on the running kernel (e.g. CPU affecting default optims etc) ? If that's the case you'll see it in the final .config.
Quoted 2 lines
I'll try simply rebooting the machine on each git bisect step. That should eliminate one more aspect of local state.
Just to be avoid wasting your time, perform a cold reboot (reset button). If you just do a hot reboot and the problem persists, there will still be a tiny part of doubt leaving a "what if" in your mind that will make you want to run it all again.
Willy