Skip to contentSkip to conversation

license: Improve disk usage

PATCH In review Richard Purdie 3 participants 5 messages Sep 12, 2014 – Sep 15, 2014

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 license: Improve disk usage

Currently copies of the license files are made which wastes disk space and adversely affects performance. We can link these instead in most cases for small performance gains.

meta/classes/license.bbclass +8 -1
index 601f561..a34ea39 100644
@@ -145,7 +145,14 @@ def copy_license_files(lic_files_paths, destdir):
     bb.utils.mkdirhier(destdir)
     for (basename, path) in lic_files_paths:
         try:
-            ret = shutil.copyfile(path, os.path.join(destdir, basename))
+            src = path
+            dst = os.path.join(destdir, basename)
+            if os.path.exists(dst):
+                os.remove(dst)
+            if (os.stat(src).st_dev == os.stat(destdir).st_dev):
+                os.link(src, dst)
+            else:
+                shutil.copyfile(src, dst)
         except Exception as e:
             bb.warn("Could not copy license file %s: %s" % (basename, e))
 
#1 lore raw

I can probably come up with scenarios where "os.stat(src).st_dev == os.stat(destdir).st_dev" but os.link will still fail.

Less code and does not assume that when src and destdir are on the same device, they can be linked:

try:
os.link(src, dst)
except:
shutil.copyfile(src, dst)

Mike.

Quoted 18 lines · Richard Purdie

Currently copies of the license files are made which wastes disk space and adversely affects performance. We can link these instead in most cases for small performance gains.

meta/classes/license.bbclass +1 -1
index 601f561..a34ea39 100644
@@ -145,7 +145,14 @@ def copy_license_files(lic_files_paths, destdir):
      bb.utils.mkdirhier(destdir)
      for (basename, path) in lic_files_paths:
          try:
-            ret = shutil.copyfile(path, os.path.join(destdir, basename))
+            src = path

meta/classes/license.bbclass def copy_license_files(lic_files_paths, destdir): · line 158

Quoted 10 lines
meta/classes/license.bbclass +7 -0
+            dst = os.path.join(destdir, basename)
+            if os.path.exists(dst):
+                os.remove(dst)
+            if (os.stat(src).st_dev == os.stat(destdir).st_dev):
+                os.link(src, dst)
+            else:
+                shutil.copyfile(src, dst)
          except Exception as e:
              bb.warn("Could not copy license file %s: %s" % (basename, e))
 

Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems Eindhovenseweg 32-C, NL-5683 KH Best Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax: (+31) (0) 499 33 69 70
E-mail: [email protected]
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

#2 lore raw
Quoted 10 lines · Mike Looijmans

I can probably come up with scenarios where "os.stat(src).st_dev == os.stat(destdir).st_dev" but os.link will still fail.

Less code and does not assume that when src and destdir are on the same device, they can be linked:

try:
os.link(src, dst)
except:
shutil.copyfile(src, dst)

I agree that is probably easier. I'm curious which scenarios you're thinking of though. We do have that device comparison elsewhere in the codebase and I copied it mainly as it was how it was done elsewhere...

Cheers,

Richard

#3 lore raw
Quoted 7 lines · Richard Purdie
Quoted 6 lines · Mike Looijmans

I can probably come up with scenarios where "os.stat(src).st_dev == os.stat(destdir).st_dev" but os.link will still fail.

Less code and does not assume that when src and destdir are on the same device, they can be linked:

Quoted 9 lines
Quoted 4 lines

try:
os.link(src, dst)
except:
shutil.copyfile(src, dst)

I agree that is probably easier. I'm curious which scenarios you're thinking of though. We do have that device comparison elsewhere in the codebase and I copied it mainly as it was how it was done elsewhere...

Simplest case is when the underlying filesystem does not support hardlinks.

I admit that chances of that being the case here will be very very small.

It's just that the try/except version will always work on any system, whereas the device comparison may fail on some setups. And given the habit of programmers to copy these things from other locations, chances are that a device comparison will some day be used on the "downloads" directory which is FAT on my system.

Also, when the copy fails, the resulting message should be clearer, as the "copyfile" call will be in the stacktrace.

Mike.

Met vriendelijke groet / kind regards,

Mike Looijmans

TOPIC Embedded Systems Eindhovenseweg 32-C, NL-5683 KH Best Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax: (+31) (0) 499 33 69 70
E-mail: [email protected]
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

#4 lore raw
Quoted 8 lines · Mike Looijmans

It's just that the try/except version will always work on any system, whereas the device comparison may fail on some setups. And given the habit of programmers to copy these things from other locations, chances are that a device comparison will some day be used on the "downloads" directory which is FAT on my system.

Also, when the copy fails, the resulting message should be clearer, as the "copyfile" call will be in the stacktrace.

I believe this is called the "try and ask forgiveness" pattern. It's race-free, too.

Ross