Fix file checksum failure with broken symlinks

If using OE's externalsrc with a source tree that is not tracked by git
and contains broken symlinks, you can receive "TypeError: unorderable
types: NoneType() < str()" within the file checksum code due to:

checksums.sort(key=operator.itemgetter(1))

Don't add files with no checksum to the checksums list in order to avoid
this.

Signed-off-by Paul Eggleton <[email protected]>

---
 lib/bb/checksum.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

lib/bb/checksum.py +4 -2

index be4ab68..8428920 100644
@@ -120,13 +120,15 @@ class FileChecksumCache(MultiProcessCache):
                             checksums.extend(checksum_dir(f))
                     else:
                         checksum = checksum_file(f)
-                        checksums.append((f, checksum))
+                        if checksum:
+                            checksums.append((f, checksum))
             elif os.path.isdir(pth):
                 if not os.path.islink(pth):
                     checksums.extend(checksum_dir(pth))
             else:
                 checksum = checksum_file(pth)
-                checksums.append((pth, checksum))
+                if checksum:
+                    checksums.append((pth, checksum))
 
         checksums.sort(key=operator.itemgetter(1))
         return checksums

Signature · 1 line

2.5.5

Quoted 6 lines · Paul Eggleton

If using OE's externalsrc with a source tree that is not tracked by git and contains broken symlinks, you can receive "TypeError: unorderable

types: NoneType() < str()" within the file checksum code due to:
checksums.sort(key=operator.itemgetter(1))

Quoted 2 lines

Don't add files with no checksum to the checksums list in order to avoid this.

In some of the cache code we have to note whether files were checked for but didn't exist. This means that if they then do exist in a later run, we know to invalidate the cache.

Does this change break that?

Quoted 8 lines · On Tue, 26 Jul 2016 08:08:42 Richard Purdie wrote:

Quoted 7 lines · Paul Eggleton

If using OE's externalsrc with a source tree that is not tracked by git and contains broken symlinks, you can receive "TypeError:
unorderable

types: NoneType() < str()" within the file checksum code due to:
checksums.sort(key=operator.itemgetter(1))

Quoted 8 lines

Quoted 2 lines

Don't add files with no checksum to the checksums list in order to avoid this.

In some of the cache code we have to note whether files were checked for but didn't exist. This means that if they then do exist in a later run, we know to invalidate the cache.

Does this change break that?

Took me a couple of days to get back to this, but after double-checking both the code and the behaviour the answer is no. As you may recall the file checksum operation is broken into two parts:

1) Getting the list of files regardless of whether they exist or not, which is done at parse time and goes into the cache

2) Actually getting the checksum of each file that does exist, which is done when we calculate the task hashes. The hashes are also cached in their own separate cache file based on mtime.