Skip to contentSkip to conversation

fs/squashfs: honour the number of parallel jobs

PATCH Superseded Yann E. MORIN 3 participants 5 messages Aug 6, 2017 – Aug 9, 2017 SquashFS file system

lore
Share & tools
lore

Yann E. MORIN

author
#0 lore raw

PATCH v2 1/1 fs/squashfs: honour the number of parallel jobs

By default, mksquashfs uses all the CPUs available, like us with BR2_JLEVEL=0.

However, by setting BR21_JLEVEL to non-zero, a user volutarily limits the number of jobbs allowed to run concurrently (for various reasons, like this is not a priority task on the machine; or like the autobuilders, to spread the load accross multiple runs without overloading the machine).

So, propagate tthe parallelisn settings down to mksquashfs, to repect the user's will.

Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Clayton Shotwell <[email protected]>

---
Changes v1 -> v2:
- enhance commit log (Thomas)

---
 fs/squashfs/squashfs.mk | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
fs/squashfs/squashfs.mk +3 -2
index 342c7a8a37..7de7f51af1 100644
@@ -6,6 +6,8 @@
 
 ROOTFS_SQUASHFS_DEPENDENCIES = host-squashfs
 
+ROOTFS_SQUASHFS_ARGS = -noappend -processors $(PARALLEL_JOBS)
+
 ifeq ($(BR2_TARGET_ROOTFS_SQUASHFS4_LZ4),y)
 ROOTFS_SQUASHFS_ARGS += -comp lz4 -Xhc
 else ifeq ($(BR2_TARGET_ROOTFS_SQUASHFS4_LZO),y)

@@ -19,8 +21,7 @@ ROOTFS_SQUASHFS_ARGS += -comp gzip
 endif
 
 define ROOTFS_SQUASHFS_CMD
-	$(HOST_DIR)/bin/mksquashfs $(TARGET_DIR) $@ -noappend \
-		$(ROOTFS_SQUASHFS_ARGS)
+	$(HOST_DIR)/bin/mksquashfs $(TARGET_DIR) $@ $(ROOTFS_SQUASHFS_ARGS)
 endef
 
 $(eval $(call ROOTFS_TARGET,squashfs))
Signature · 1 line
2.11.0
#1 lore raw

PATCH v2 1/1 fs/squashfs: honour the number of parallel jobs

Yann,

Quoted 3 lines · Yann E. MORIN

By default, mksquashfs uses all the CPUs available, like us with BR2_JLEVEL=0.

This ends up using the number of CPUs available plus 1. I tested this on my machine and it worked. I do still see the extended attributes issue that I posted
in https://bugs.busybox.net/show_bug.cgi?id=10141.

Quoted 11 lines

However, by setting BR21_JLEVEL to non-zero, a user volutarily limits the number of jobbs allowed to run concurrently (for various reasons, like this is not a priority task on the machine; or like the autobuilders, to spread the load accross multiple runs without overloading the machine).

So, propagate tthe parallelisn settings down to mksquashfs, to repect the user's will.

Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Clayton Shotwell <[email protected]>
Tested-by Clayton Shotwell <[email protected]>

Thanks,
Clayton

#2 lore raw

PATCH v2 1/1 fs/squashfs: honour the number of parallel jobs

Quoted 18 lines · Clayton Shotwell

Yann,

Quoted 3 lines · Yann E. MORIN

By default, mksquashfs uses all the CPUs available, like us with BR2_JLEVEL=0.

This ends up using the number of CPUs available plus 1. I tested this on my machine and it worked. I do still see the extended attributes issue that I posted
in https://bugs.busybox.net/show_bug.cgi?id=10141.

Quoted 6 lines

However, by setting BR21_JLEVEL to non-zero, a user volutarily limits the number of jobbs allowed to run concurrently (for various reasons, like this is not a priority task on the machine; or like the autobuilders, to spread the load accross multiple runs without overloading the machine).

Quoted 7 lines
Quoted 5 lines

So, propagate tthe parallelisn settings down to mksquashfs, to repect the user's will.

Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Clayton Shotwell <[email protected]>
Tested-by Clayton Shotwell <[email protected]>

Thanks. I'm not sure if I should apply this to next or to master. It doesn't
really fix anything, so I'm inclined to put it in next...

For the real mksquashfs issue, do we have a way to find out what goes wrong
exactly? From what I've read, it's not even clear if it's mksquashfs or fakeroot that is at fault...

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
author
#3 lore raw

PATCH v2 1/1 fs/squashfs: honour the number of parallel jobs

Arnout, All,

On 2017-08-08 16:09 +0200, Arnout Vandecappelle spake thusly:

Quoted 19 lines · Clayton Shotwell
Quoted 18 lines

Yann,

Quoted 3 lines · Yann E. MORIN

By default, mksquashfs uses all the CPUs available, like us with BR2_JLEVEL=0.

This ends up using the number of CPUs available plus 1. I tested this on my machine and it worked. I do still see the extended attributes issue that I posted
in https://bugs.busybox.net/show_bug.cgi?id=10141.

Quoted 6 lines

However, by setting BR21_JLEVEL to non-zero, a user volutarily limits the number of jobbs allowed to run concurrently (for various reasons, like this is not a priority task on the machine; or like the autobuilders, to spread the load accross multiple runs without overloading the machine).

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

So, propagate tthe parallelisn settings down to mksquashfs, to repect the user's will.

Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Clayton Shotwell <[email protected]>
Tested-by Clayton Shotwell <[email protected]>

Thanks!

Quoted 2 lines

Thanks. I'm not sure if I should apply this to next or to master. It doesn't
really fix anything, so I'm inclined to put it in next...

Yes, next is good.

Quoted 3 lines

For the real mksquashfs issue, do we have a way to find out what goes wrong
exactly? From what I've read, it's not even clear if it's mksquashfs or fakeroot that is at fault...

So, the first analysis points toward highly-parallel situations.

I'll do some experimentation...

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
Applied
#4 lore raw

PATCH v2 1/1 fs/squashfs: honour the number of parallel jobs

In reply to Yann E. MORIN
Quoted 9 lines · Yann E. MORIN

By default, mksquashfs uses all the CPUs available, like us with BR2_JLEVEL=0.

However, by setting BR21_JLEVEL to non-zero, a user volutarily limits the number of jobbs allowed to run concurrently (for various reasons, like this is not a priority task on the machine; or like the autobuilders, to spread the load accross multiple runs without overloading the machine).

Quoted 5 lines

So, propagate tthe parallelisn settings down to mksquashfs, to repect the user's will.

Signed-off-by "Yann E. MORIN" <[email protected]>
Cc Clayton Shotwell <[email protected]>

Applied to next, thanks.

Regards,
Arnout

Signature · 6 lines
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF