Fix random kernel unpack failure

PATCH In review Yu Ke 4 participants 6 messages Jan 4, 2011 – Jan 5, 2011

Share & tools

Yu Ke

author

2011-01-04 11:27 UTC #0 lore raw

cover Fix random kernel unpack failure

autobuilder recently is blocked by random unpack failure of linux-libc-headers-yocto and linux-yocto. this patch fix this issue. the patch description has the detailed root cause and fix.

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/kernel-unpack-fix
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/kernel-unpack-fix

Thanks,
Yu Ke <[email protected]>
---

Yu Ke (1):
base.bbclass: add lock file for do_unpack task

 meta/classes/base.bbclass |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

Yu Ke

author

2011-01-04 11:28 UTC #1 lore raw

PATCH 1/1 base.bbclass: add lock file for do_unpack task

This patch intend to fix the random unpack failure of linux-libc-headers-yocto and linux-yocto.

The root cause of the unpack failure is that: these two recpies has the same URL, thus has the same dest file during the fetch and unpack phase:
do_fetch : create tar ball ${DL_DIR}/git_git.pokylinux.org.linux-yocto-2.6.37.tar.gz do_unpack : extract tar ball ${DL_DIR}/git_git.pokylinux.org.linux-yocto-2.6.37.tar.gz fetch phase is protected by lockfile, so it works fine. but unpack phase is not lock protected, thus there is race condition like: when linux-yocto do_unpack is extracting the tar ball, linux-libc-headers-yocto do_fetch starts to create tar ball thus overwrite linux-yocto's tar ball and cause linux-yocto do_unpack failure

To fix this issue, do_unpack also need to be protected by lock

Signed-off-by Yu Ke <[email protected]>

---
 meta/classes/base.bbclass |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

meta/classes/base.bbclass +4 -0

index 54a1dcd..f8ce123 100644
@@ -242,6 +242,8 @@ python base_do_unpack() {
 	localdata = bb.data.createCopy(d)
 	bb.data.update_data(localdata)
 
+	urldata = bb.fetch.init([], localdata, True)
+
 	src_uri = bb.data.getVar('SRC_URI', localdata, True)
 	if not src_uri:
 		return

@@ -253,7 +255,9 @@ python base_do_unpack() {
 		if local is None:
 			continue
 		local = os.path.realpath(local)
+		lf = bb.utils.lockfile(urldata[url].lockfile)
 		ret = oe_unpack_file(local, localdata, url)
+		bb.utils.unlockfile(lf)
 		if not ret:
 			raise bb.build.FuncFailed("oe_unpack_file failed with return value %s" % ret)
 }

Signature · 1 line

1.7.0.4

Darren Hart

2011-01-04 15:45 UTC #2 lore raw

Re: [PATCH 1/1] base.bbclass: add lock file for do_unpack task

Quoted 12 lines · Yu Ke

This patch intend to fix the random unpack failure of linux-libc-headers-yocto and linux-yocto.
The root cause of the unpack failure is that: these two recpies has the same URL, thus has the same dest file during the fetch and unpack phase:
do_fetch : create tar ball ${DL_DIR}/git_git.pokylinux.org.linux-yocto-2.6.37.tar.gz do_unpack : extract tar ball ${DL_DIR}/git_git.pokylinux.org.linux-yocto-2.6.37.tar.gz fetch phase is protected by lockfile, so it works fine. but unpack phase is not lock protected, thus there is race condition like: when linux-yocto do_unpack is extracting the tar ball, linux-libc-headers-yocto do_fetch starts to create tar ball thus overwrite linux-yocto's tar ball and cause linux-yocto do_unpack failure

Quoted 1 line

To fix this issue, do_unpack also need to be protected by lock

Nice work! My out-for-review trace-cmd and kernelshark recipes will suffer the same issue.

Signature · 2 lines

Darren Hart
Yocto Linux Kernel

Saul Wold

2011-01-04 19:52 UTC #4 lore raw

In reply to Yu Ke

Quoted 16 lines · Yu Ke

autobuilder recently is blocked by random unpack failure of linux-libc-headers-yocto and linux-yocto. this patch fix this issue. the patch description has the detailed root cause and fix.
Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/kernel-unpack-fix
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/kernel-unpack-fix
Thanks,
Yu Ke<[email protected]>
---
Yu Ke (1):
base.bbclass: add lock file for do_unpack task

Quoted 8 lines

  meta/classes/base.bbclass |    4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)
_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

Ke,

This has been pulled into master as a workaround for the existing issue, please ensure that this issue gets addressed properly once the fetch work is completed to a point that this can be done.

Thanks
Sau!

Yu Ke

author

2011-01-05 01:52 UTC #5 lore raw

Quoted 26 lines · Saul Wold

Quoted 16 lines · Yu Ke
autobuilder recently is blocked by random unpack failure of linux-libc-headers-yocto and linux-yocto. this patch fix this issue. the patch description has the detailed root cause and fix.
Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: kyu3/kernel-unpack-fix
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=kyu3/kernel-unpack-fix
Thanks,
Yu Ke<[email protected]>
---
Yu Ke (1):
base.bbclass: add lock file for do_unpack task
Quoted 8 lines
 meta/classes/base.bbclass |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)
_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

Quoted 8 lines

Ke,
This has been pulled into master as a workaround for the existing issue, please ensure that this issue gets addressed properly once the fetch work is completed to a point that this can be done.
Thanks
Sau!

Yes, with the new fetcher, do_fetch will not create the tar ball for git repo, instead, do_unpack will directly checkout the code from the git repo, thus there will be no race condition and the lock can be safely removed.

I will take care of it in the new fetcher.

Regards
Ke

Quoted 4 lines

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

Cui, Dexuan

2011-01-05 02:18 UTC #3 lore raw

Re: [PATCH 1/1] base.bbclass: add lock file for do_unpack task

In reply to Darren Hart

Quoted 19 lines · Darren Hart

Quoted 18 lines · Yu Ke
This patch intend to fix the random unpack failure of linux-libc-headers-yocto
and linux-yocto.
The root cause of the unpack failure is that: these two recpies has the same URL, thus
has the same dest file during the fetch and unpack phase:
do_fetch : create tar ball ${DL_DIR}/git_git.pokylinux.org.linux-yocto-2.6.37.tar.gz do_unpack : extract tar ball ${DL_DIR}/git_git.pokylinux.org.linux-yocto-2.6.37.tar.gz fetch phase is protected by lockfile, so it works fine. but unpack phase is not lock protected, thus there is race condition like: when linux-yocto do_unpack is extracting the tar ball, linux-libc-headers-yocto do_fetch starts to create tar ball thus overwrite linux-yocto's
tar ball and cause linux-yocto do_unpack failure

Quoted 4 lines

Quoted 1 line
To fix this issue, do_unpack also need to be protected by lock
Nice work! My out-for-review trace-cmd and kernelshark recipes will suffer the same issue.

Thanks to Ke's fix!
Previously I met with the linux-libc-headers unpack failure everytime... Now the failure has gone with the fix.

Thanks,
-- Dexuan