Skip to contentSkip to conversation

state/staging: Fix directory not deleted for race

PATCH Superseded Pgowda 4 participants 14 messages Sep 15, 2021 – Oct 19, 2021 2 messages not in the archive

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Pgowda

author
#0 lore raw

PATCH 1/1 state/staging: Fix directory not deleted for race

The race issue resulted in some empty directories not being deleted. During staging_copyfile, it results in "file already exists error" on creating link for directories during do_prepare_recipe_sysroot. The following patch checks whether the directory and its sub directories have no files in it and removes them.

The issue was observed on running rust-hello-world for glibc and musl one after the other.
TCLIBC=glibc MACHINE=qemuarm bitbake rust-hello-world TCLIBC=musl MACHINE=qemuarm bitbake rust-hello-world

On fixing the above issue, it was found that arm-poky-linux-musleabi.json files are not being generated. $/del/build/tmp/work/x86_64-linux/rust-cross-arm/1.54.0-r0/targets/ arm-poky-linux-musleabi.json': No such file or directory

It is generated and builds successfully when musl lib is built standalone. However, when musl is built after glibc or viceversa results in the latter json file not being generated.

---
 meta/classes/staging.bbclass | 10 ++++++++++
 1 file changed, 10 insertions(+)
meta/classes/staging.bbclass +10 -0
index af3397bab6..4b751af4d3 100644
@@ -145,6 +145,16 @@ def staging_copyfile(c, target, dest, postinsts, seendirs):
     if os.path.islink(c):
         linkto = os.readlink(c)
         if os.path.lexists(dest):
+            # Check if directory is not deleted for race issue
+            if os.path.isdir(dest):
+                count = 0
+                for root, dirs, files in os.walk(dest):
+                    for Files in files:
+                        count += 1
+                if count == 0:
+                    import shutil
+                    shutil.rmtree(dest)
+                    os.symlink(linkto, dest)
             if not os.path.islink(dest):
                 raise OSError(errno.EEXIST, "Link %s already exists as a file" % dest, dest)
             if os.readlink(dest) == linkto:
Signature · 1 line
2.31.1
#1 lore raw
Quoted 36 lines · Pgowda

The race issue resulted in some empty directories not being deleted. During staging_copyfile, it results in "file already exists error" on creating link for directories during do_prepare_recipe_sysroot. The following patch checks whether the directory and its sub directories have no files in it and removes them.

The issue was observed on running rust-hello-world for glibc and musl one after the other.
TCLIBC=glibc MACHINE=qemuarm bitbake rust-hello-world TCLIBC=musl MACHINE=qemuarm bitbake rust-hello-world

On fixing the above issue, it was found that arm-poky-linux-musleabi.json files are not being generated. $/del/build/tmp/work/x86_64-linux/rust-cross-arm/1.54.0-r0/targets/ arm-poky-linux-musleabi.json': No such file or directory

It is generated and builds successfully when musl lib is built standalone. However, when musl is built after glibc or viceversa results in the latter json file not being generated.

---
 meta/classes/staging.bbclass | 10 ++++++++++
 1 file changed, 10 insertions(+)
meta/classes/staging.bbclass +3 -0
index af3397bab6..4b751af4d3 100644
@@ -145,6 +145,16 @@ def staging_copyfile(c, target, dest, postinsts, seendirs):
     if os.path.islink(c):
         linkto = os.readlink(c)
         if os.path.lexists(dest):
+            # Check if directory is not deleted for race issue
+            if os.path.isdir(dest):
+                count = 0

meta/classes/staging.bbclass def staging_copyfile(c, target, dest, postinsts, seendirs): · line 160

Quoted 10 lines
meta/classes/staging.bbclass +7 -0
+                for root, dirs, files in os.walk(dest):
+                    for Files in files:
+                        count += 1
+                if count == 0:
+                    import shutil
+                    shutil.rmtree(dest)
+                    os.symlink(linkto, dest)
             if not os.path.islink(dest):
                 raise OSError(errno.EEXIST, "Link %s already exists as a file" % dest, dest)
             if os.readlink(dest) == linkto:

This change looks like a good fix but it can be improved a little.

Firstly, rather than the count and os.walk, we can just use "if not os.listdir(dest):".

Secondly, if we know the directory is empty and is a directory, we don' need rmtree, we can just use os.rmdir() which should be much faster.

I'd also clarify the comment to something like:

# Empty directories are not removed due to race issues but if we

# find a conflict, we can remove empty directories here.

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 53 lines · Richard Purdie via lists.openembedded.org
Quoted 36 lines · Pgowda

The race issue resulted in some empty directories not being deleted. During staging_copyfile, it results in "file already exists error" on creating link for directories during do_prepare_recipe_sysroot. The following patch checks whether the directory and its sub directories have no files in it and removes them.

The issue was observed on running rust-hello-world for glibc and musl one after the other.
TCLIBC=glibc MACHINE=qemuarm bitbake rust-hello-world TCLIBC=musl MACHINE=qemuarm bitbake rust-hello-world

On fixing the above issue, it was found that arm-poky-linux-musleabi.json files are not being generated. $/del/build/tmp/work/x86_64-linux/rust-cross-arm/1.54.0-r0/targets/ arm-poky-linux-musleabi.json': No such file or directory

It is generated and builds successfully when musl lib is built standalone. However, when musl is built after glibc or viceversa results in the latter json file not being generated.

---
 meta/classes/staging.bbclass | 10 ++++++++++
 1 file changed, 10 insertions(+)
meta/classes/staging.bbclass +3 -0
index af3397bab6..4b751af4d3 100644
@@ -145,6 +145,16 @@ def staging_copyfile(c, target, dest, postinsts, seendirs):
     if os.path.islink(c):
         linkto = os.readlink(c)
         if os.path.lexists(dest):
+            # Check if directory is not deleted for race issue
+            if os.path.isdir(dest):
+                count = 0

meta/classes/staging.bbclass def staging_copyfile(c, target, dest, postinsts, seendirs): · line 160

Quoted 10 lines
meta/classes/staging.bbclass +7 -0
+                for root, dirs, files in os.walk(dest):
+                    for Files in files:
+                        count += 1
+                if count == 0:
+                    import shutil
+                    shutil.rmtree(dest)
+                    os.symlink(linkto, dest)
             if not os.path.islink(dest):
                 raise OSError(errno.EEXIST, "Link %s already exists as a file" % dest, dest)
             if os.readlink(dest) == linkto:

This change looks like a good fix but it can be improved a little.

Firstly, rather than the count and os.walk, we can just use "if not os.listdir(dest):".

Quoted 7 lines

Secondly, if we know the directory is empty and is a directory, we don' need rmtree, we can just use os.rmdir() which should be much faster.

I'd also clarify the comment to something like:

# Empty directories are not removed due to race issues but if we

# find a conflict, we can remove empty directories here.

Looking further at this and your description, I think you might be saying there are multiple levels of empty directories. If that is the case my simplification may not work but I am more worried about why we have multiple levels of this issue.

I would also like to understand why these files overlap and are symlinks in one case but directories in another. Can you show some specific examples of the files in question?

The missing json file sounds like there could be some file not being handled by sstate but it is hard to know without looking at the code.

Cheers,

Richard

author
#4 lore raw

Hi Richard,

Thanks for reviewing the patch and your comments.

Quoted 2 lines

Looking further at this and your description, I think you might be saying there are multiple levels of empty directories.

Yes. There are multiple level of directories as follows:-

/tmp/work/x86_64-linux/rust-cross-x86_64/1.54.0-r0/recipe-sysroot\
/usr/include/c++/11.2.0/x86_64-poky-linux/ext /tmp/work/x86_64-linux/rust-cross-x86_64/1.54.0-r0/recipe-sysroot\
/usr/include/c++/11.2.0/x86_64-poky-linux/bits /tmp/work/x86_64-linux/rust-cross-x86_64/1.54.0-r0/recipe-sysroot\
/usr/include/c++/11.2.0/x86_64-poky-linux

Quoted 3 lines

I would also like to understand why these files overlap and are symlinks in one case but directories in another. Can you show some specific examples of the files in question?

It happens when we build rust-hello-world for glibc followed by musl as follows:-
TCLIBC=glibc MACHINE=qemuarm bitbake rust-hello-world TCLIBC=musl MACHINE=qemuarm bitbake rust-hello-world

The recipe do_prepare_recipe_sysroot tries to populate the sysroot file. The glibc build had already generated the sysroot and copied properly. When we build the musl later on, it tries to remove the above directory and symlink
the newly built musl files. It was deleted properly before the race issue was observed.
However, after the race issue; files are being deleted but the directory and sub directories
are left undeleted which resulted in the error.

Quoted 2 lines

The missing json file sounds like there could be some file not being handled by sstate but it is hard to know without looking at the code.

Thanks. I will try to debug the issue further.

Regards,
pgowda

#5 lore raw

Add Khem (aka Mr Musl). ;-)

Quoted 25 lines · Pgowda

Hi Richard,

Thanks for reviewing the patch and your comments.

Quoted 2 lines

Looking further at this and your description, I think you might be saying there are multiple levels of empty directories.

Yes. There are multiple level of directories as follows:-

/tmp/work/x86_64-linux/rust-cross-x86_64/1.54.0-r0/recipe-sysroot\
/usr/include/c++/11.2.0/x86_64-poky-linux/ext /tmp/work/x86_64-linux/rust-cross-x86_64/1.54.0-r0/recipe-sysroot\
/usr/include/c++/11.2.0/x86_64-poky-linux/bits /tmp/work/x86_64-linux/rust-cross-x86_64/1.54.0-r0/recipe-sysroot\
/usr/include/c++/11.2.0/x86_64-poky-linux

Quoted 3 lines

I would also like to understand why these files overlap and are symlinks in one case but directories in another. Can you show some specific examples of the files in question?

It happens when we build rust-hello-world for glibc followed by musl as follows:-
TCLIBC=glibc MACHINE=qemuarm bitbake rust-hello-world TCLIBC=musl MACHINE=qemuarm bitbake rust-hello-world

Quoted 9 lines

The recipe do_prepare_recipe_sysroot tries to populate the sysroot file. The glibc build had already generated the sysroot and copied properly. When we build the musl later on, it tries to remove the above directory and symlink
the newly built musl files. It was deleted properly before the race issue was observed.
However, after the race issue; files are being deleted but the directory and sub directories
are left undeleted which resulted in the error.

All of that makes sense to me based on first reading it but I'm not intimately familiar with toolchain sysroot setup.

Quoted 5 lines
Quoted 3 lines

The missing json file sounds like there could be some file not being handled by sstate but it is hard to know without looking at the code.

Thanks. I will try to debug the issue further.

So do you have additional information Naveen or
do you maintain that the fix presented on this thread is all that is needed.

../Randy

Quoted 14 lines

Regards,
pgowda

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156105): https://lists.openembedded.org/g/openembedded-core/message/156105
Mute This Topic: https://lists.openembedded.org/mt/85625195/3616765
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

Signature · 2 lines
# Randy MacLeod
# Wind River Linux
#6 lore raw
In reply to Pgowda
Quoted 35 lines · pgowda cve

Hi Richard,

Thanks for reviewing the patch and your comments.

Quoted 2 lines

Looking further at this and your description, I think you might be saying there are multiple levels of empty directories.

Yes. There are multiple level of directories as follows:-

/tmp/work/x86_64-linux/rust-cross-x86_64/1.54.0-r0/recipe-sysroot\
/usr/include/c++/11.2.0/x86_64-poky-linux/ext /tmp/work/x86_64-linux/rust-cross-x86_64/1.54.0-r0/recipe-sysroot\
/usr/include/c++/11.2.0/x86_64-poky-linux/bits /tmp/work/x86_64-linux/rust-cross-x86_64/1.54.0-r0/recipe-sysroot\
/usr/include/c++/11.2.0/x86_64-poky-linux

Quoted 3 lines

I would also like to understand why these files overlap and are symlinks in one case but directories in another. Can you show some specific examples of the files in question?

It happens when we build rust-hello-world for glibc followed by musl as follows:-
TCLIBC=glibc MACHINE=qemuarm bitbake rust-hello-world TCLIBC=musl MACHINE=qemuarm bitbake rust-hello-world

The recipe do_prepare_recipe_sysroot tries to populate the sysroot file. The glibc build had already generated the sysroot and copied properly. When we build the musl later on, it tries to remove the above directory and symlink
the newly built musl files. It was deleted properly before the race issue was observed.
However, after the race issue; files are being deleted but the directory and sub directories
are left undeleted which resulted in the error.

Quoted 4 lines
Quoted 2 lines

The missing json file sounds like there could be some file not being handled by sstate but it is hard to know without looking at the code.

Thanks. I will try to debug the issue further.

I did think about this a bit further. "cross" recipes are hard and I suspect in this case we may need to account for musl/glibc in the recipe name or it's workdir if it is going to be "picky" like this.

Cheers,

Richard

author
#7 lore raw

PATCH v2 1/1 state/staging: Fix directory not deleted for race

Modify the SYSROOT PATH to fix race around issue when rust-hello-world is run for two libc one after the other.

---
 meta/recipes-devtools/rust/rust-cross.inc | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
meta/recipes-devtools/rust/rust-cross.inc +11 -1
index bee7c9f12f..09d0078d17 100644
@@ -32,7 +32,17 @@ DEPENDS += "virtual/${TARGET_PREFIX}gcc virtual/${TARGET_PREFIX}compilerlibs vir
 DEPENDS += "rust-native"
 
 PROVIDES = "virtual/${TARGET_PREFIX}rust"
-PN = "rust-cross-${TARGET_ARCH}"
+
+# Modify the sysroot PATH based on the libc used to fix race around issue.
+def get_libc(d):
+    tclibc = d.getVar('TCLIBC')
+    if tclibc == 'glibc':
+        pn = "rust-cross-${TARGET_ARCH}"
+    else:
+        pn = "rust-cross-${TARGET_ARCH}-${TCLIBC}"
+    return pn
+
+PN := "${@get_libc(d)}"
 
 # In the cross compilation case, rustc doesn't seem to get the rpath quite
 # right. It manages to include '../../lib/${TARGET_PREFIX}', but doesn't
Signature · 1 line
2.31.1
#8 lore raw
Quoted 19 lines · Pgowda

Modify the SYSROOT PATH to fix race around issue when rust-hello-world is run for two libc one after the other.

---
 meta/recipes-devtools/rust/rust-cross.inc | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
meta/recipes-devtools/rust/rust-cross.inc +2 -1
index bee7c9f12f..09d0078d17 100644
@@ -32,7 +32,17 @@ DEPENDS += "virtual/${TARGET_PREFIX}gcc virtual/${TARGET_PREFIX}compilerlibs vir
 DEPENDS += "rust-native"
 
 PROVIDES = "virtual/${TARGET_PREFIX}rust"
-PN = "rust-cross-${TARGET_ARCH}"
+
+# Modify the sysroot PATH based on the libc used to fix race around issue.

meta/recipes-devtools/rust/rust-cross.inc DEPENDS += "virtual/${TARGET_PREFIX}gcc virtual/${TARGET_PREFIX}compilerlibs vir · line 46

Quoted 10 lines
meta/recipes-devtools/rust/rust-cross.inc +9 -0
+def get_libc(d):
+    tclibc = d.getVar('TCLIBC')
+    if tclibc == 'glibc':
+        pn = "rust-cross-${TARGET_ARCH}"
+    else:
+        pn = "rust-cross-${TARGET_ARCH}-${TCLIBC}"
+    return pn
+
+PN := "${@get_libc(d)}"
 

I'm wondering if we should just add TCLIBC to PN unconditionally? Do we need the complexity of the conditional code?

Cheers,

Richard

author
#9 lore raw

Hi Richard,

Thanks for the review and suggestion.

Quoted 1 line
Quoted 1 line

I'm wondering if we should just add TCLIBC to PN unconditionally?

I didn't want to add TCLIBC to the code when using default GLIBC. Hence, I tried to keep the folder default for GLIBC.

However, if you suggest adding TCLIBC to PN unconditionally; I will modify the patch and submit.

Thanks,
pgowda

#10 lore raw
Quoted 11 lines · pgowda cve

Hi Richard,

Thanks for the review and suggestion.

Quoted 1 line
Quoted 1 line

I'm wondering if we should just add TCLIBC to PN unconditionally?

I didn't want to add TCLIBC to the code when using default GLIBC. Hence, I tried to keep the folder default for GLIBC.

However, if you suggest adding TCLIBC to PN unconditionally; I will modify the patch and submit.

Lets try that please.

There is a related issue Martin reported where the recipe is really TUNE_ARCH specific and not really TARGET_ARCH specific. One step at a time I guess though.

Cheers,

Richard

#11 lore raw
Quoted 13 lines · Richard Purdie
Quoted 11 lines · pgowda cve

Hi Richard,

Thanks for the review and suggestion.

Quoted 1 line
Quoted 1 line

I'm wondering if we should just add TCLIBC to PN unconditionally?

I didn't want to add TCLIBC to the code when using default GLIBC. Hence, I tried to keep the folder default for GLIBC.

However, if you suggest adding TCLIBC to PN unconditionally; I will modify the patch and submit.

Quoted 7 lines

Lets try that please.

There is a related issue Martin reported where the recipe is really TUNE_ARCH
specific and not really TARGET_ARCH specific. One step at a time I guess though.

I think you were referring to:
https://lists.openembedded.org/g/openembedded-core/message/156516
and it's currently even MACHINE_ARCH due to poking in MACHINEOVERRIDES variable, but in theory it should be possible to re-write the logic to be only TUNE_ARCH specific.

Cheers,

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#15 lore raw

Hi,

Gentle ping on this patch.

Thanks,
pgowda

Quoted 23 lines · Pgowda via lists.openembedded.org

Hi Richard,

Quoted 2 lines
Quoted 2 lines

we may need to account for musl/glibc in the recipe name or it's workdir if it is going to be "picky" like this.

Thanks for your suggestion on fixing this issue in a better way.

Please find the patch posted at the following link as per your suggestion and let me know if it's fine.
https://lists.openembedded.org/g/openembedded-core/message/156513

Thanks,
pgowda

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#156514): https://lists.openembedded.org/g/openembedded-core/message/156514
Mute This Topic: https://lists.openembedded.org/mt/85625195/6402864
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-