Skip to contentSkip to conversation

toaster: Add fake entry to Target_File for filesystem root

PATCH In review Elliot Smith 3 participants 6 messages Sep 21, 2015 – Sep 23, 2015

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 toaster: Add fake entry to Target_File for filesystem root

The files-in-image.txt file is produced by bitbake after an image is created, listing all the files in the image. However, this list doesn't include the root directory ('/').

buildinfohelper.py then tries to construct the filesystem tree from this file, assuming that every directory apart from the root directory (which is special-cased) can be assigned a parent. But because the root directory isn't listed in files-in-image.txt, an object for the root directory is never created.

The direct subdirectories of the root ('./bin', './usr' etc.) then can't be assigned a parent directory, as the object representing the root directory doesn't exist. This results in a Target_File lookup error and causes the directory listing page to fail.

Fix this by creating a fake entry for the root directory in the Target_File table, so that the direct subdirectories of / can be assigned a parent. Note that it doesn't matter that the root is faked, as its properties are never shown in the directory structure tree.

[YOCTO #8280]

---
 bitbake/lib/bb/ui/buildinfohelper.py | 31 +++++++++++++++++--------------
 1 file changed, 17 insertions(+), 14 deletions(-)
bitbake/lib/bb/ui/buildinfohelper.py +17 -14
index 6e313fe..b2201f6 100644
@@ -353,26 +353,29 @@ class ORMWrapper(object):
         files = filedata['files']
         syms = filedata['syms']
 
-        # we insert directories, ordered by name depth
+        # always create the root directory as a special case;
+        # note that this is never displayed, so the owner, group,
+        # size, permission are irrelevant
+        tf_obj = Target_File.objects.create(target = target_obj,
+                                            path = '/',
+                                            size = 0,
+                                            owner = '',
+                                            group = '',
+                                            permission = '',
+                                            inodetype = Target_File.ITYPE_DIRECTORY)
+        tf_obj.save()
+
+        # insert directories, ordered by name depth
         for d in sorted(dirs, key=lambda x:len(x[-1].split("/"))):
             (user, group, size) = d[1:4]
             permission = d[0][1:]
             path = d[4].lstrip(".")
+
+            # we already created the root directory, so ignore any
+            # entry for it
             if len(path) == 0:
-                # we create the root directory as a special case
-                path = "/"
-                tf_obj = Target_File.objects.create(
-                        target = target_obj,
-                        path = path,
-                        size = size,
-                        inodetype = Target_File.ITYPE_DIRECTORY,
-                        permission = permission,
-                        owner = user,
-                        group = group,
-                        )
-                tf_obj.directory = tf_obj
-                tf_obj.save()
                 continue
+
             parent_path = "/".join(path.split("/")[:len(path.split("/")) - 1])
             if len(parent_path) == 0:
                 parent_path = "/"

--
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

#1 lore raw

upstreamed.
ty,
b

Quoted 99 lines · Elliot Smith

The files-in-image.txt file is produced by bitbake after an image is created, listing all the files in the image. However, this list doesn't include the root directory ('/').

buildinfohelper.py then tries to construct the filesystem tree from this file, assuming that every directory apart from the root directory (which is special-cased) can be assigned a parent. But because the root directory isn't listed in files-in-image.txt, an object for the root directory is never created.

The direct subdirectories of the root ('./bin', './usr' etc.) then can't be assigned a parent directory, as the object representing the root directory doesn't exist. This results in a Target_File lookup error and causes the directory listing page to fail.

Fix this by creating a fake entry for the root directory in the Target_File table, so that the direct subdirectories of / can be assigned a parent. Note that it doesn't matter that the root is faked, as its properties are never shown in the directory structure tree.

[YOCTO #8280]

---
 bitbake/lib/bb/ui/buildinfohelper.py | 31 +++++++++++++++++--------------
 1 file changed, 17 insertions(+), 14 deletions(-)
bitbake/lib/bb/ui/buildinfohelper.py +17 -15
index 6e313fe..b2201f6 100644
@@ -353,26 +353,29 @@ class ORMWrapper(object):
         files = filedata['files']
         syms = filedata['syms']
 
-        # we insert directories, ordered by name depth
+        # always create the root directory as a special case;
+        # note that this is never displayed, so the owner, group,
+        # size, permission are irrelevant
+        tf_obj = Target_File.objects.create(target = target_obj,
+                                            path = '/',
+                                            size = 0,
+                                            owner = '',
+                                            group = '',
+                                            permission = '',
+                                            inodetype = Target_File.ITYPE_DIRECTORY)
+        tf_obj.save()
+
+        # insert directories, ordered by name depth
         for d in sorted(dirs, key=lambda x:len(x[-1].split("/"))):
             (user, group, size) = d[1:4]
             permission = d[0][1:]
             path = d[4].lstrip(".")
+
+            # we already created the root directory, so ignore any
+            # entry for it
             if len(path) == 0:
-                # we create the root directory as a special case
-                path = "/"
-                tf_obj = Target_File.objects.create(
-                        target = target_obj,
-                        path = path,
-                        size = size,
-                        inodetype = Target_File.ITYPE_DIRECTORY,
-                        permission = permission,
-                        owner = user,
-                        group = group,
-                        )
-                tf_obj.directory = tf_obj
-                tf_obj.save()
                 continue
+
             parent_path = "/".join(path.split("/")[:len(path.split("/")) - 1])
             if len(parent_path) == 0:
                 parent_path = "/"
--

Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster

#2 lore raw
In reply to Elliot Smith
Quoted 4 lines · Elliot Smith

The files-in-image.txt file is produced by bitbake after an image is created, listing all the files in the image. However, this list doesn't include the root directory ('/').

Did you try to look what caused this change? Bitbake used to include root dir into this file previously, right?

Quoted 13 lines

buildinfohelper.py then tries to construct the filesystem tree from this file, assuming that every directory apart from the root directory (which is special-cased) can be assigned a parent. But because the root directory isn't listed in files-in-image.txt, an object for the root directory is never created.

The direct subdirectories of the root ('./bin', './usr' etc.) then can't be assigned a parent directory, as the object representing the root directory doesn't exist. This results in a Target_File lookup error and causes the directory listing page to fail.

Quoted 6 lines

Fix this by creating a fake entry for the root directory in the Target_File table, so that the direct subdirectories of / can be assigned a parent. Note that it doesn't matter that the root is faked, as its properties are never shown in the directory structure tree.

This looks like a workaround to me. It should be fixed in bitbake code if it's possible.

Regards,
Ed

Quoted 76 lines

[YOCTO #8280]

---
 bitbake/lib/bb/ui/buildinfohelper.py | 31 +++++++++++++++++--------------
 1 file changed, 17 insertions(+), 14 deletions(-)
bitbake/lib/bb/ui/buildinfohelper.py +17 -15
index 6e313fe..b2201f6 100644
@@ -353,26 +353,29 @@ class ORMWrapper(object):
         files = filedata['files']
         syms = filedata['syms']
 
-        # we insert directories, ordered by name depth
+        # always create the root directory as a special case;
+        # note that this is never displayed, so the owner, group,
+        # size, permission are irrelevant
+        tf_obj = Target_File.objects.create(target = target_obj,
+                                            path = '/',
+                                            size = 0,
+                                            owner = '',
+                                            group = '',
+                                            permission = '',
+                                            inodetype = Target_File.ITYPE_DIRECTORY)
+        tf_obj.save()
+
+        # insert directories, ordered by name depth
         for d in sorted(dirs, key=lambda x:len(x[-1].split("/"))):
             (user, group, size) = d[1:4]
             permission = d[0][1:]
             path = d[4].lstrip(".")
+
+            # we already created the root directory, so ignore any
+            # entry for it
             if len(path) == 0:
-                # we create the root directory as a special case
-                path = "/"
-                tf_obj = Target_File.objects.create(
-                        target = target_obj,
-                        path = path,
-                        size = size,
-                        inodetype = Target_File.ITYPE_DIRECTORY,
-                        permission = permission,
-                        owner = user,
-                        group = group,
-                        )
-                tf_obj.directory = tf_obj
-                tf_obj.save()
                 continue
+
             parent_path = "/".join(path.split("/")[:len(path.split("/")) - 1])
             if len(parent_path) == 0:
                 parent_path = "/"
--

Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

Signature · 4 lines
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster
Signature · 3 lines
--
Regards,
Ed
#3 lore raw
Quoted 8 lines · Ed Bartosh
Quoted 4 lines · Elliot Smith

The files-in-image.txt file is produced by bitbake after an image is created, listing all the files in the image. However, this list doesn't include the root directory ('/').

Did you try to look what caused this change? Bitbake used to include root dir into this file previously, right?

I'm not sure if it ever did include this file. I had a brief look at how buildhistory_analysis.py does its work (git diffs if I read it correctly), but didn't dig right into it.

Quoted 13 lines
Quoted 13 lines

buildinfohelper.py then tries to construct the filesystem tree from this file, assuming that every directory apart from the root directory (which is special-cased) can be assigned a parent. But because the root directory isn't listed in files-in-image.txt, an object for the root directory is never created.

The direct subdirectories of the root ('./bin', './usr' etc.) then can't be assigned a parent directory, as the object representing the root directory doesn't exist. This results in a Target_File lookup error and causes the directory listing page to fail.

Quoted 9 lines
Quoted 6 lines

Fix this by creating a fake entry for the root directory in the Target_File table, so that the direct subdirectories of / can be assigned a parent. Note that it doesn't matter that the root is faked, as its properties are never shown in the directory structure tree.

This looks like a workaround to me. It should be fixed in bitbake code if it's possible.

I discussed this with Brian, and the decision was that a work-around was
sufficient for now. (See the options I suggested in
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8280#c9.)

I can investigate this some more and/or raise a bug, if we think that's worthwhile.

Elliot

Signature · 3 lines
Elliot Smith
Software Engineer
Intel Open Source Technology Centre
#4 lore raw

yes, please raise the buildhistory bug with bitbake but for now, I'm ok with this workaround, especially since you did the sensible thing of checking to see if it exists before creating your entry i.e. this code will still work even after/if they fix the buildhistory bug. -b

Quoted 59 lines · Elliot
Quoted 8 lines · Ed Bartosh
Quoted 4 lines · Elliot Smith

The files-in-image.txt file is produced by bitbake after an image is created, listing all the files in the image. However, this list doesn't include the root directory ('/').

Did you try to look what caused this change? Bitbake used to include root dir into this file previously, right?

I'm not sure if it ever did include this file. I had a brief look at how buildhistory_analysis.py does its work (git diffs if I read it correctly), but didn't dig right into it.

Quoted 14 lines
Quoted 14 lines

buildinfohelper.py then tries to construct the filesystem tree from this file, assuming that every directory apart from the root directory (which is special-cased) can be assigned a parent. But because the root directory isn't listed in files-in-image.txt, an object for the root directory is never created.

The direct subdirectories of the root ('./bin', './usr' etc.) then can't be assigned a parent directory, as the object representing the root directory doesn't exist. This results in a Target_File lookup error and causes the directory listing page to fail.

Quoted 8 lines
Quoted 6 lines

Fix this by creating a fake entry for the root directory in the Target_File table, so that the direct subdirectories of / can be assigned a parent. Note that it doesn't matter that the root is faked, as its properties are never shown in the directory structure tree.

This looks like a workaround to me. It should be fixed in bitbake code if it's possible.

I discussed this with Brian, and the decision was that a work-around was
sufficient for now. (See the options I suggested in
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8280#c9.)

I can investigate this some more and/or raise a bug, if we think that's worthwhile.

Elliot
--
Elliot Smith
Software Engineer
Intel Open Source Technology Centre

--
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster

#5 lore raw
Quoted 5 lines · Brian Avery

yes, please raise the buildhistory bug with bitbake but for now, I'm ok with this workaround, especially since you did the sensible thing of checking to see if it exists before creating your entry i.e. this code will still work even after/if they fix the buildhistory bug.

I raised a bug for the buildhistory issue:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8370

Elliot

Signature · 3 lines
Elliot Smith
Software Engineer
Intel Open Source Technology Centre