kernel.bbclass: make KERNEL_DEBUG_TIMESTAMPS work at rebuild
PATCH v2 1/1 kernel.bbclass: make KERNEL_DEBUG_TIMESTAMPS work at rebuild
Currently, the KERNEL_DEBUG_TIMESTAMPS is not working as expected at rebuild. That is, even if we set it to "1", the kernel build time is not changed. The problem could be reproduced by the following steps.
1. bitbake core-image-minimal; start image and check uname -a output.
2. set in local.conf: KERNEL_DEBUG_TIMESTAMPS = "1"
3. bitbake core-image-minimal; start image and check uname -a output.
It's expected that after enabling KERNEL_DEBUG_TIMESTAMPS, the kernel build time will be set to current date. But it's not. This is because the compile.h was not re-generated when do_compile task was re-executed.
In mkcompile_h, we have:
"""
# Only replace the real compile.h if the new one is different,
# in order to preserve the timestamp and avoid unnecessary # recompilations. # We don't consider the file changed if only the date/time changed, # unless KBUILD_BUILD_TIMESTAMP was explicitly set (e.g. for # reproducible builds with that value referring to a commit timestamp). # A kernel config change will increase the generation number, thus # causing compile.h to be updated (including date/time) due to the # changed comment in the # first line. """ It has made it very clear that it will not be re-generated unless we have KBUILD_BUILD_TIMESTAMP set explicitly. So we set this variable explicitly in do_compile to fix this issue.
--- meta/classes-recipe/kernel.bbclass | 8 ++++++++ 1 file changed, 8 insertions(+)
meta/classes-recipe/kernel.bbclass +8 -0
@@ -367,6 +367,10 @@ kernel_do_compile() { export KBUILD_BUILD_TIMESTAMP="$ts" export KCONFIG_NOTIMESTAMP=1 bbnote "KBUILD_BUILD_TIMESTAMP: $ts" + else + ts=`LC_ALL=C date` + export KBUILD_BUILD_TIMESTAMP="$ts" + bbnote "KBUILD_BUILD_TIMESTAMP: $ts" fi # The $use_alternate_initrd is only set from # do_bundle_initramfs() This variable is specifically for the
@@ -412,6 +416,10 @@ do_compile_kernelmodules() { export KBUILD_BUILD_TIMESTAMP="$ts" export KCONFIG_NOTIMESTAMP=1 bbnote "KBUILD_BUILD_TIMESTAMP: $ts" + else + ts=`LC_ALL=C date` + export KBUILD_BUILD_TIMESTAMP="$ts" + bbnote "KBUILD_BUILD_TIMESTAMP: $ts" fi if (grep -q -i -e '^CONFIG_MODULES=y$' ${B}/.config); then oe_runmake -C ${B} ${PARALLEL_MAKE} modules ${KERNEL_EXTRA_ARGS}
Signature · 1 line
2.17.1
Chen Qi <[email protected]> escreveu no dia quinta, 17/11/2022 à(s) 04:12:
Quoted 60 lines
Currently, the
KERNEL_DEBUG_TIMESTAMPSis not working as expected at rebuild. That is, even if we set it to "1", the kernel build time is not changed. The problem could be reproduced by the following steps.
1. bitbake core-image-minimal; start image and checkuname -aoutput.
2. set in local.conf:KERNEL_DEBUG_TIMESTAMPS= "1"
3. bitbake core-image-minimal; start image and checkuname -aoutput.It's expected that after enabling
KERNEL_DEBUG_TIMESTAMPS, the kernel build time will be set to current date. But it's not. This is because thecompile.hwas not re-generated whendo_compiletask was re-executed.In
mkcompile_h, we have:
"""
# Only replace the realcompile.hif the new one is different,# in order to preserve the timestamp and avoid unnecessary # recompilations. # We don't consider the file changed if only the date/time changed, # unless KBUILD_BUILD_TIMESTAMP was explicitly set (e.g. for # reproducible builds with that value referring to a commit timestamp). # A kernel config change will increase the generation number, thus # causing compile.h to be updated (including date/time) due to the # changed comment in the # first line. """ It has made it very clear that it will not be re-generated unless we have KBUILD_BUILD_TIMESTAMP set explicitly. So we set this variable explicitly in do_compile to fix this issue.Signed-off-by Chen Qi <[email protected]>--- meta/classes-recipe/kernel.bbclass | 8 ++++++++ 1 file changed, 8 insertions(+)meta/classes-recipe/kernel.bbclass +8
-0@@ -367,6 +367,10 @@ kernel_do_compile() { export KBUILD_BUILD_TIMESTAMP="$ts" export KCONFIG_NOTIMESTAMP=1 bbnote "KBUILD_BUILD_TIMESTAMP: $ts" + else + ts=`LC_ALL=C date` + export KBUILD_BUILD_TIMESTAMP="$ts" + bbnote "KBUILD_BUILD_TIMESTAMP: $ts" fi # The $use_alternate_initrd is only set from # do_bundle_initramfs() This variable is specifically for the@@ -412,6 +416,10 @@ do_compile_kernelmodules() { export KBUILD_BUILD_TIMESTAMP="$ts" export KCONFIG_NOTIMESTAMP=1 bbnote "KBUILD_BUILD_TIMESTAMP: $ts" + else + ts=`LC_ALL=C date` + export KBUILD_BUILD_TIMESTAMP="$ts" + bbnote "KBUILD_BUILD_TIMESTAMP: $ts" fi if (grep -q -i -e '^CONFIG_MODULES=y$' ${B}/.config); then oe_runmake -C ${B} ${PARALLEL_MAKE} modules
Quoted 4 lines
${KERNEL_EXTRA_ARGS}
--
2.17.1
Hi Chen,
I think this only works on the first time you run it and after that if the kernel can be taken from the sstate-cache this timestamp will be bypassed. To generate a new timestamp you need to invalidate the stamp and run the task again with:
bitbake virtual/linux -C do_compile
Another possible issue is this will make this task indeterministic (not reproducible) as it uses the command 'date' that generates a different output in each run.
IMO the bitbake DATETIME is better and it is deterministic:
export KBUILD_BUILD_TIMESTAMP="$DATETIME"
This also maybe introduces a new variable dependency exclusion on the tasks that uses this variable:
kernel_do_compile[vardepsexclude] += DATETIME do_compile_kernelmodules[vardepsexclude] += DATETIME
Jose
Quoted 12 lines
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#173401):
https://lists.openembedded.org/g/openembedded-core/message/173401
Mute This Topic: https://lists.openembedded.org/mt/95083712/5052612
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Signature · 3 lines
Best regards, José Quaresma
Hi Jose,
Thanks a lot for your review. I’d like to explain the case in more details.
* indeterministic (not reproducible) issue
The use case is: the user wants to disable reproducible build for kernel, and wants to get the actual build time of kernel. In other words, we want it to be indeterministic. This is actually what KERNEL_DEBUG_TIMESTAMPS is expected to do.
* state cache issue
I think if the kernel is taken from the sstate cache and not rebuilt, it’s acceptable. The timestamp in /proc/version (or uname -a output) is reflecting the actual kernel build time. The situation that really confuses users is that when kernel do_compile is re-executed, the timestamp is not updated, even if we want it to (by enabling KERNEL_DEBUG_TIMESTAMPS). It’s complained that even do_compile is forced to re-run, the timestamp is not updated.
In such case, I only see two choices:
1. delete the compile.h in do_compile (I have to admit that I didn’t try this method out)
2. set KBUILD_BUILD_TIMESTAMP explicitly
I chose the second one because I wanted to output the value of KBUILD_BUILD_TIMESTAMP just like when KERNEL_DEBUG_TIMESTAMPS is disabled.
Also, date will be run in one way or another. It will either be run in mkcompile_h or be run in do_compile.
* DATETIME
The DATETIME’s value is the time when bitbake.conf is parsed. By using DATETIME + varexclude, it’s basically achieving the same result as using date directly.
The ssate cache issue you mentioned remains the same, although I think it’s actually the expected behavior as explained above. And when source code/config changes and do_compile is re-run, KBUILD_BUILD_TIMESTAMP is set to a different value from the previous run. The only difference is that DATETIME reflects the time bitbake.conf is parsed, and date reflects the time kernel’s do_compile is executed.
Does my explanation make some sense? If I still miss something, please let me know. Thanks again for your review. Have a good weekend 😊
Regards,
Qi
From: Jose Quaresma <[email protected]>
Sent: Friday, November 18, 2022 7:11 PM
To: Chen, Qi <[email protected]>
Cc: [email protected]
Subject: Re: [OE-core][master][kirkstone][PATCH V2] kernel.bbclass: make KERNEL_DEBUG_TIMESTAMPS work at rebuild
Chen Qi <[email protected]<mailto:[email protected]>> escreveu no dia quinta, 17/11/2022 à(s) 04:12:
Currently, the KERNEL_DEBUG_TIMESTAMPS is not working as expected
at rebuild. That is, even if we set it to "1", the kernel build time
is not changed. The problem could be reproduced by the following steps.
1. bitbake core-image-minimal; start image and check uname -a output.
2. set in local.conf: KERNEL_DEBUG_TIMESTAMPS = "1"
3. bitbake core-image-minimal; start image and check uname -a output.
It's expected that after enabling KERNEL_DEBUG_TIMESTAMPS, the kernel build time will be set to current date. But it's not. This is because the compile.h was not re-generated when do_compile task was re-executed.
In mkcompile_h, we have:
"""
# Only replace the real compile.h if the new one is different,
# in order to preserve the timestamp and avoid unnecessary # recompilations. # We don't consider the file changed if only the date/time changed, # unless KBUILD_BUILD_TIMESTAMP was explicitly set (e.g. for # reproducible builds with that value referring to a commit timestamp). # A kernel config change will increase the generation number, thus # causing compile.h to be updated (including date/time) due to the # changed comment in the # first line. """ It has made it very clear that it will not be re-generated unless we have KBUILD_BUILD_TIMESTAMP set explicitly. So we set this variable explicitly in do_compile to fix this issue.
--- meta/classes-recipe/kernel.bbclass | 8 ++++++++ 1 file changed, 8 insertions(+)
meta/classes-recipe/kernel.bbclass +8 -0
@@ -367,6 +367,10 @@ kernel_do_compile() { export KBUILD_BUILD_TIMESTAMP="$ts" export KCONFIG_NOTIMESTAMP=1 bbnote "KBUILD_BUILD_TIMESTAMP: $ts" + else + ts=`LC_ALL=C date` + export KBUILD_BUILD_TIMESTAMP="$ts" + bbnote "KBUILD_BUILD_TIMESTAMP: $ts" fi # The $use_alternate_initrd is only set from # do_bundle_initramfs() This variable is specifically for the
@@ -412,6 +416,10 @@ do_compile_kernelmodules() { export KBUILD_BUILD_TIMESTAMP="$ts" export KCONFIG_NOTIMESTAMP=1 bbnote "KBUILD_BUILD_TIMESTAMP: $ts" + else + ts=`LC_ALL=C date` + export KBUILD_BUILD_TIMESTAMP="$ts" + bbnote "KBUILD_BUILD_TIMESTAMP: $ts" fi if (grep -q -i -e '^CONFIG_MODULES=y$' ${B}/.config); then oe_runmake -C ${B} ${PARALLEL_MAKE} modules ${KERNEL_EXTRA_ARGS}
--
2.17.1
Hi Chen,
I think this only works on the first time you run it and after that if the kernel can be taken from the sstate-cache this timestamp will be bypassed. To generate a new timestamp you need to invalidate the stamp and run the task again with:
bitbake virtual/linux -C do_compile
Another possible issue is this will make this task indeterministic (not reproducible) as it uses the command 'date' that generates a different output in each run.
IMO the bitbake DATETIME is better and it is deterministic:
export KBUILD_BUILD_TIMESTAMP="$DATETIME"
This also maybe introduces a new variable dependency exclusion on the tasks that uses this variable:
kernel_do_compile[vardepsexclude] += DATETIME do_compile_kernelmodules[vardepsexclude] += DATETIME
Jose
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#173401): https://lists.openembedded.org/g/openembedded-core/message/173401<https://urldefense.com/v3/__https:/lists.openembedded.org/g/openembedded-core/message/173401__;!!AjveYdw8EvQ!fB6p0AH069Vo-AVee7wFlAtEYf7mk5iseFIxw2QmL7Zr3UD9hP_vy_yAKs2N-InQPVv6rWGLb7ZaCmGE7pHmHz74$>
Mute This Topic: https://lists.openembedded.org/mt/95083712/5052612<https://urldefense.com/v3/__https:/lists.openembedded.org/mt/95083712/5052612__;!!AjveYdw8EvQ!fB6p0AH069Vo-AVee7wFlAtEYf7mk5iseFIxw2QmL7Zr3UD9hP_vy_yAKs2N-InQPVv6rWGLb7ZaCmGE7nBibunR$>
Group Owner: [email protected]<mailto:openembedded-core%[email protected]>
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub<https://urldefense.com/v3/__https:/lists.openembedded.org/g/openembedded-core/unsub__;!!AjveYdw8EvQ!fB6p0AH069Vo-AVee7wFlAtEYf7mk5iseFIxw2QmL7Zr3UD9hP_vy_yAKs2N-InQPVv6rWGLb7ZaCmGE7qAoD1Gm$> [[email protected]<mailto:[email protected]>]
-=-=-=-=-=-=-=-=-=-=-=-
--
Best regards,
José Quaresma
Hi Qi,
Chen, Qi <[email protected]> escreveu no dia sexta, 18/11/2022 à(s) 14:06:
Quoted 8 lines
Hi Jose,
Thanks a lot for your review. I’d like to explain the case in more details.
Quoted 7 lines
* indeterministic (not reproducible) issue
The use case is: the user wants to disable reproducible build for kernel, and wants to get the actual build time of kernel. In other words, we want it to be indeterministic. This is actually what
KERNEL_DEBUG_TIMESTAMPSis expected to do.
I don't fully understand the use case when looking at the patch as one of the primary intentions is make it not reproducible.
Quoted 8 lines
* state cache issue
I think if the kernel is taken from the sstate cache and not rebuilt, it’s acceptable. The timestamp in
/proc/version(oruname -aoutput) is reflecting the actual kernel build time.
Quoted 7 lines
The situation that really confuses users is that when kernel
do_compileis re-executed, the timestamp is not updated, even if we want it to (by enablingKERNEL_DEBUG_TIMESTAMPS).It’s complained that even
do_compileis forced to re-run, the timestamp is not updated.
Understand so the main idea is to get the timestamp of the last do_compile task running time even when they will come from the sstate, when it comes from the sstate the timestamp is the same one that was taken when the do_compile runs.
Quoted 21 lines
In such case, I only see two choices:
1. delete the
compile.hindo_compile(I have to admit that I didn’t try this method out)2. set
KBUILD_BUILD_TIMESTAMPexplicitlyI chose the second one because I wanted to output the value of
KBUILD_BUILD_TIMESTAMPjust like whenKERNEL_DEBUG_TIMESTAMPSis disabled.Also,
datewill be run in one way or another. It will either be run inmkcompile_hor be run indo_compile.* DATETIME
The DATETIME’s value is the time when bitbake.conf is parsed. By using DATETIME + varexclude, it’s basically achieving the same result as using
datedirectly.
Quoted 8 lines
The ssate cache issue you mentioned remains the same, although I think it’s actually the expected behavior as explained above. And when source code/config changes and
do_compileis re-run,KBUILD_BUILD_TIMESTAMPis set to a different value from the previous run.The only difference is that DATETIME reflects the time bitbake.conf is parsed, and
datereflects the time kernel’sdo_compileis executed.
This suggestion is only to be more bitbake friendly and use a variable that already exists, using data will provide the same and is maybe better in this case as it does not need the varexclude.
Quoted 5 lines
Does my explanation make some sense? If I still miss something, please let me know.
Thanks for explaining the full picture.
Best regards,
Jose
Quoted 147 lines
Thanks again for your review. Have a good weekend 😊
Regards,
Qi
*From:* Jose Quaresma <[email protected]>
*Sent:* Friday, November 18, 2022 7:11 PM
*To:* Chen, Qi <[email protected]>
*Cc:* [email protected]
*Subject:* Re: [OE-core][master][kirkstone][PATCH V2] kernel.bbclass:
makeKERNEL_DEBUG_TIMESTAMPSwork at rebuildChen Qi <[email protected]> escreveu no dia quinta, 17/11/2022 à(s) 04:12:
Currently, the
KERNEL_DEBUG_TIMESTAMPSis not working as expected at rebuild. That is, even if we set it to "1", the kernel build time is not changed. The problem could be reproduced by the following steps.
1. bitbake core-image-minimal; start image and checkuname -aoutput.
2. set in local.conf:KERNEL_DEBUG_TIMESTAMPS= "1"
3. bitbake core-image-minimal; start image and checkuname -aoutput.It's expected that after enabling
KERNEL_DEBUG_TIMESTAMPS, the kernel build time will be set to current date. But it's not. This is because thecompile.hwas not re-generated whendo_compiletask was re-executed.In
mkcompile_h, we have:
"""
# Only replace the realcompile.hif the new one is different,# in order to preserve the timestamp and avoid unnecessary # recompilations. # We don't consider the file changed if only the date/time changed, # unless KBUILD_BUILD_TIMESTAMP was explicitly set (e.g. for # reproducible builds with that value referring to a commit timestamp). # A kernel config change will increase the generation number, thus # causing compile.h to be updated (including date/time) due to the # changed comment in the # first line. """ It has made it very clear that it will not be re-generated unless we have KBUILD_BUILD_TIMESTAMP set explicitly. So we set this variable explicitly in do_compile to fix this issue.Signed-off-by Chen Qi <[email protected]>--- meta/classes-recipe/kernel.bbclass | 8 ++++++++ 1 file changed, 8 insertions(+)meta/classes-recipe/kernel.bbclass +8
-0@@ -367,6 +367,10 @@ kernel_do_compile() { export KBUILD_BUILD_TIMESTAMP="$ts" export KCONFIG_NOTIMESTAMP=1 bbnote "KBUILD_BUILD_TIMESTAMP: $ts" + else + ts=`LC_ALL=C date` + export KBUILD_BUILD_TIMESTAMP="$ts" + bbnote "KBUILD_BUILD_TIMESTAMP: $ts" fi # The $use_alternate_initrd is only set from # do_bundle_initramfs() This variable is specifically for the@@ -412,6 +416,10 @@ do_compile_kernelmodules() { export KBUILD_BUILD_TIMESTAMP="$ts" export KCONFIG_NOTIMESTAMP=1 bbnote "KBUILD_BUILD_TIMESTAMP: $ts" + else + ts=`LC_ALL=C date` + export KBUILD_BUILD_TIMESTAMP="$ts" + bbnote "KBUILD_BUILD_TIMESTAMP: $ts" fi if (grep -q -i -e '^CONFIG_MODULES=y$' ${B}/.config); then oe_runmake -C ${B} ${PARALLEL_MAKE} modules${KERNEL_EXTRA_ARGS}
--
2.17.1Hi Chen,
I think this only works on the first time you run it and after that if the kernel can be taken from the sstate-cache this timestamp will be bypassed. To generate a new timestamp you need to invalidate the stamp and run the task again with:
bitbake virtual/linux -C
do_compileAnother possible issue is this will make this task indeterministic (not reproducible) as it uses the command 'date' that generates a different output in each run.
IMO the bitbake DATETIME is better and it is deterministic:
export KBUILD_BUILD_TIMESTAMP="$DATETIME"
This also maybe introduces a new variable dependency exclusion on the tasks that uses this variable:
kernel_do_compile[vardepsexclude] += DATETIME
do_compile_kernelmodules[vardepsexclude] += DATETIME
Jose
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#173401):
https://lists.openembedded.org/g/openembedded-core/message/173401
<https://urldefense.com/v3/__https:/lists.openembedded.org/g/openembedded-core/message/173401__;!!AjveYdw8EvQ!fB6p0AH069Vo-AVee7wFlAtEYf7mk5iseFIxw2QmL7Zr3UD9hP_vy_yAKs2N-InQPVv6rWGLb7ZaCmGE7pHmHz74$>
Mute This Topic: https://lists.openembedded.org/mt/95083712/5052612
<https://urldefense.com/v3/__https:/lists.openembedded.org/mt/95083712/5052612__;!!AjveYdw8EvQ!fB6p0AH069Vo-AVee7wFlAtEYf7mk5iseFIxw2QmL7Zr3UD9hP_vy_yAKs2N-InQPVv6rWGLb7ZaCmGE7nBibunR$>
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
<https://urldefense.com/v3/__https:/lists.openembedded.org/g/openembedded-core/unsub__;!!AjveYdw8EvQ!fB6p0AH069Vo-AVee7wFlAtEYf7mk5iseFIxw2QmL7Zr3UD9hP_vy_yAKs2N-InQPVv6rWGLb7ZaCmGE7qAoD1Gm$>
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=---
Best regards,
José Quaresma
Signature · 3 lines
Best regards, José Quaresma