Skip to contentSkip to conversation

bitbake: cooker: Ensure bbappends are found in stable order

PATCH Peter Kjellerstedt 2 participants 2 messages Sep 11, 2019

lore
Share & tools
lore
#0 lore raw
Quoted 34 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Wes Lindauer
Sent: den 28 augusti 2019 23:38
To: [email protected]
Subject: [OE-core] [PATCH] bitbake: cooker: Ensure bbappends are found in stable order

Thanks to wildcards in bbappend filenames, it's possible to have multiple bbappends that apply to the same recipe in the same directory. In order to get sstate hits between different workspaces, we want to apply those bbappend files in a consistent order. Since readdir() returns files in a non-deterministic order between workspaces (based on inode number and/or time of creation), we'll need to sort its result in order to have any consistency.

---
 bitbake/lib/bb/cooker.py | 1 +
 1 file changed, 1 insertion(+)
bitbake/lib/bb/cooker.py +1 -1
index 0607fcc708..a9ee142aa5 100644
@@ -1869,6 +1869,7 @@ class CookerCollectFiles(object):
             (bbappend, filename) = b
             if (bbappend == f) or ('%' in bbappend and bbappend.startswith(f[:bbappend.index('%')])):
                 filelist.append(filename)
+        filelist.sort()
         return filelist
 
     def collection_priorities(self, pkgfns, d):
--
Quoted 1 line

2.14.5

I don't think this was a good idea at all. Before, bbappends were applied in the priority order of the layers, but this change throws the sorting based on priority out the window and might cause bbappend files for higher layers to be applied before lower layers' bbappend files.

Please revert this change, and rethink how to maintain a stable order for bbappend files from the same directory without affecting bbappend files from different layers.

//Peter

#1 lore raw
Quoted 41 lines · Peter Kjellerstedt
Quoted 34 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Wes Lindauer
Sent: den 28 augusti 2019 23:38
To: [email protected]
Subject: [OE-core] [PATCH] bitbake: cooker: Ensure bbappends are found in stable order

Thanks to wildcards in bbappend filenames, it's possible to have multiple bbappends that apply to the same recipe in the same directory. In order to get sstate hits between different workspaces, we want to apply those bbappend files in a consistent order. Since readdir() returns files in a non-deterministic order between workspaces (based on inode number and/or time of creation), we'll need to sort its result in order to have any consistency.

---
 bitbake/lib/bb/cooker.py | 1 +
 1 file changed, 1 insertion(+)
bitbake/lib/bb/cooker.py +1 -1
index 0607fcc708..a9ee142aa5 100644
@@ -1869,6 +1869,7 @@ class CookerCollectFiles(object):
             (bbappend, filename) = b
             if (bbappend == f) or ('%' in bbappend and bbappend.startswith(f[:bbappend.index('%')])):
                 filelist.append(filename)
+        filelist.sort()
         return filelist
 
     def collection_priorities(self, pkgfns, d):
--
Quoted 1 line

2.14.5

I don't think this was a good idea at all. Before, bbappends were applied in the priority order of the layers, but this change throws the sorting based on priority out the window and might cause bbappend files for higher layers to be applied before lower layers' bbappend files.

Quoted 3 lines

Please revert this change, and rethink how to maintain a stable order for bbappend files from the same directory without affecting bbappend files from different layers.

I had assumed that the sorting was only within each layer's bbappends. If it's global, you are absolutely right that priority order matters.

So maybe:

Within a layer (multiple .bbappends) 'sort' (alphabetic)

Between layers first by priority, then by order in the bblayers (for two layers with the same priority)?

--Mark

Quoted 2 lines

//Peter