Skip to contentSkip to conversation

Current strategy is to use right/left shift to implement div/mult

RFC PATCH New Mauro Condarelli 4 participants 9 messages Sep 30, 2020 – Oct 1, 2020 SquashFS file system

lore
Share & tools
lore
#0 lore raw

v2 cover Current strategy is to use right/left shift to implement div/mult

this work *only* under the assumption block-size is always a power of 2; I was unable to find where it is enforced.

Alternative strategy would be to use directly the lower-level do_div(), or the lldiv() wrapper instead.

Please review because I did not find a place where it is enforced that struct blk_desc blksz is a a power of 2 and thus using log2blksz is actually correct.

Changes in v2:
- replace division with right shift (Daniel Schwierzeck).
- remove vocore2-specific change (Daniel Schwierzeck).
- add warning to Kconfig about CONFIG_SYS_MALLOC_LEN (Tom Rini).

Mauro Condarelli (1):
Fix missing __udivdi3 in SquashFS implementation.

 fs/squashfs/Kconfig      |  2 ++
 fs/squashfs/sqfs.c       | 30 +++++++++++++++---------------
 fs/squashfs/sqfs_inode.c |  6 +++---
 3 files changed, 20 insertions(+), 18 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

RFC PATCH v2 1/1 Fix missing __udivdi3 in SquashFS implementation.

Use right shift to avoid 64-bit divisions.

These divisions are needed to convert from file length (potentially over 32-bit range) to block number, so result and remainder are guaranteed to fit in 32-bit integers.

---

Changes in v2:
- replace division with right shift (Daniel Schwierzeck).
- remove vocore2-specific change (Daniel Schwierzeck).
- add warning to Kconfig about CONFIG_SYS_MALLOC_LEN (Tom Rini).

 fs/squashfs/Kconfig      |  2 ++
 fs/squashfs/sqfs.c       | 30 +++++++++++++++---------------
 fs/squashfs/sqfs_inode.c |  6 +++---
 3 files changed, 20 insertions(+), 18 deletions(-)
fs/squashfs/Kconfig +2 -0
index 54ab1618f1..7c3f83d007 100644
@@ -9,3 +9,5 @@ config FS_SQUASHFS
 	  filesystem use, for archival use (i.e. in cases where a .tar.gz file
 	  may be used), and in constrained block device/memory systems (e.g.
 	  embedded systems) where low overhead is needed.
+	  WARNING: if compression is enabled SquashFS needs a large amount
+	  of dynamic memory; make sure CONFIG_SYS_MALLOC_LEN >= 0x4000.
fs/squashfs/sqfs.c +15 -15
index 15208b4dab..90e6848e6c 100644
@@ -85,10 +85,10 @@ static int sqfs_calc_n_blks(__le64 start, __le64 end, u64 *offset)
 	u64 start_, table_size;
 
 	table_size = le64_to_cpu(end) - le64_to_cpu(start);
-	start_ = le64_to_cpu(start) / ctxt.cur_dev->blksz;
-	*offset = le64_to_cpu(start) - (start_ * ctxt.cur_dev->blksz);
+	start_ = le64_to_cpu(start) >> ctxt.cur_dev->log2blksz;
+	*offset = le64_to_cpu(start) - (start_ << ctxt.cur_dev->log2blksz);
 
-	return DIV_ROUND_UP(table_size + *offset, ctxt.cur_dev->blksz);
+	return (table_size + *offset + ctxt.cur_dev->blksz - 1) >> ctxt.cur_dev->log2blksz;
 }
 
 /*
@@ -109,8 +109,8 @@ static int sqfs_frag_lookup(u32 inode_fragment_index,
 	if (inode_fragment_index >= get_unaligned_le32(&sblk->fragments))
 		return -EINVAL;
 
-	start = get_unaligned_le64(&sblk->fragment_table_start) /
-		ctxt.cur_dev->blksz;
+	start = get_unaligned_le64(&sblk->fragment_table_start) >>
+		ctxt.cur_dev->log2blksz;
 	n_blks = sqfs_calc_n_blks(sblk->fragment_table_start,
 				  sblk->export_table_start,
 				  &table_offset);
@@ -135,7 +135,7 @@ static int sqfs_frag_lookup(u32 inode_fragment_index,
 	start_block = get_unaligned_le64(table + table_offset + block *
 					 sizeof(u64));
 
-	start = start_block / ctxt.cur_dev->blksz;
+	start = start_block >> ctxt.cur_dev->log2blksz;
 	n_blks = sqfs_calc_n_blks(cpu_to_le64(start_block),
 				  sblk->fragment_table_start, &table_offset);
 
@@ -641,8 +641,8 @@ static int sqfs_read_inode_table(unsigned char **inode_table)
 
 	table_size = get_unaligned_le64(&sblk->directory_table_start) -
 		get_unaligned_le64(&sblk->inode_table_start);
-	start = get_unaligned_le64(&sblk->inode_table_start) /
-		ctxt.cur_dev->blksz;
+	start = get_unaligned_le64(&sblk->inode_table_start) >>
+		ctxt.cur_dev->log2blksz;
 	n_blks = sqfs_calc_n_blks(sblk->inode_table_start,
 				  sblk->directory_table_start, &table_offset);
 
@@ -725,8 +725,8 @@ static int sqfs_read_directory_table(unsigned char **dir_table, u32 **pos_list)
 	/* DIRECTORY TABLE */
 	table_size = get_unaligned_le64(&sblk->fragment_table_start) -
 		get_unaligned_le64(&sblk->directory_table_start);
-	start = get_unaligned_le64(&sblk->directory_table_start) /
-		ctxt.cur_dev->blksz;
+	start = get_unaligned_le64(&sblk->directory_table_start) >>
+		ctxt.cur_dev->log2blksz;
 	n_blks = sqfs_calc_n_blks(sblk->directory_table_start,
 				  sblk->fragment_table_start, &table_offset);
 
@@ -1334,11 +1334,11 @@ int sqfs_read(const char *filename, void *buf, loff_t offset, loff_t len,
 	}
 
 	for (j = 0; j < datablk_count; j++) {
-		start = data_offset / ctxt.cur_dev->blksz;
+		start = data_offset >> ctxt.cur_dev->log2blksz;
 		table_size = SQFS_BLOCK_SIZE(finfo.blk_sizes[j]);
 		table_offset = data_offset - (start * ctxt.cur_dev->blksz);
-		n_blks = DIV_ROUND_UP(table_size + table_offset,
-				      ctxt.cur_dev->blksz);
+		n_blks = (table_size + table_offset + ctxt.cur_dev->blksz - 1) >>
+			ctxt.cur_dev->log2blksz;
 
 		data_buffer = malloc_cache_aligned(n_blks * ctxt.cur_dev->blksz);
 
@@ -1388,10 +1388,10 @@ int sqfs_read(const char *filename, void *buf, loff_t offset, loff_t len,
 		goto free_buffer;
 	}
 
-	start = frag_entry.start / ctxt.cur_dev->blksz;
+	start = frag_entry.start >> ctxt.cur_dev->log2blksz;
 	table_size = SQFS_BLOCK_SIZE(frag_entry.size);
 	table_offset = frag_entry.start - (start * ctxt.cur_dev->blksz);
-	n_blks = DIV_ROUND_UP(table_size + table_offset, ctxt.cur_dev->blksz);
+	n_blks = (table_size + table_offset + ctxt.cur_dev->blksz - 1) >> ctxt.cur_dev->log2blksz;
 
 	fragment = malloc_cache_aligned(n_blks * ctxt.cur_dev->blksz);
 
fs/squashfs/sqfs_inode.c +3 -3
index 1368f3063c..5dbf945c80 100644
@@ -10,7 +10,7 @@
 #include <stdint.h>
 #include <stdio.h>
 #include <stdlib.h>
-#include <string.h>
+#include <part.h>
 
 #include "sqfs_decompressor.h"
 #include "sqfs_filesystem.h"
@@ -68,9 +68,9 @@ int sqfs_inode_size(struct squashfs_base_inode *inode, u32 blk_size)
 		unsigned int blk_list_size;
 
 		if (fragment == 0xFFFFFFFF)
-			blk_list_size = DIV_ROUND_UP(file_size, blk_size);
+			blk_list_size = (file_size + blk_size - 1) >> LOG2(blk_size);
 		else
-			blk_list_size = file_size / blk_size;
+			blk_list_size = file_size > LOG2(blk_size);
 
 		return sizeof(*lreg) + blk_list_size * sizeof(u32);
 	}
Signature · 1 line
2.25.1
#2 lore raw

RFC PATCH v2 1/1 Fix missing __udivdi3 in SquashFS implementation.

Am Mittwoch, den 30.09.2020, 17:45 +0200 schrieb Mauro Condarelli:

Quoted 7 lines

Use right shift to avoid 64-bit divisions.

These divisions are needed to convert from file length (potentially over 32-bit range) to block number, so result and remainder are guaranteed to fit in 32-bit integers.

the commit subject needs to be fixed. If you change a specific subsystem, you need to add the according prefix. You can use the Git log to figure out that prefix. Also you don't fix a missing __udivdi3 , you just want to avoid 64-bit divisions.

So maybe something like: "fs/squashfs: avoid 64-bit divisions on 32-bit systems"

Quoted 124 lines

---

Changes in v2:
- replace division with right shift (Daniel Schwierzeck).
- remove vocore2-specific change (Daniel Schwierzeck).
- add warning to Kconfig about CONFIG_SYS_MALLOC_LEN (Tom Rini).

 fs/squashfs/Kconfig      |  2 ++
 fs/squashfs/sqfs.c       | 30 +++++++++++++++---------------
 fs/squashfs/sqfs_inode.c |  6 +++---
 3 files changed, 20 insertions(+), 18 deletions(-)
fs/squashfs/Kconfig +2 -0
index 54ab1618f1..7c3f83d007 100644
@@ -9,3 +9,5 @@ config FS_SQUASHFS
 	  filesystem use, for archival use (i.e. in cases where a .tar.gz file
 	  may be used), and in constrained block device/memory systems (e.g.
 	  embedded systems) where low overhead is needed.
+	  WARNING: if compression is enabled SquashFS needs a large amount
+	  of dynamic memory; make sure CONFIG_SYS_MALLOC_LEN >= 0x4000.
fs/squashfs/sqfs.c +15 -15
index 15208b4dab..90e6848e6c 100644
@@ -85,10 +85,10 @@ static int sqfs_calc_n_blks(__le64 start, __le64 end, u64 *offset)
 	u64 start_, table_size;
 
 	table_size = le64_to_cpu(end) - le64_to_cpu(start);
-	start_ = le64_to_cpu(start) / ctxt.cur_dev->blksz;
-	*offset = le64_to_cpu(start) - (start_ * ctxt.cur_dev->blksz);
+	start_ = le64_to_cpu(start) >> ctxt.cur_dev->log2blksz;
+	*offset = le64_to_cpu(start) - (start_ << ctxt.cur_dev->log2blksz);
 
-	return DIV_ROUND_UP(table_size + *offset, ctxt.cur_dev->blksz);
+	return (table_size + *offset + ctxt.cur_dev->blksz - 1) >> ctxt.cur_dev->log2blksz;
 }
 
 /*
@@ -109,8 +109,8 @@ static int sqfs_frag_lookup(u32 inode_fragment_index,
 	if (inode_fragment_index >= get_unaligned_le32(&sblk->fragments))
 		return -EINVAL;
 
-	start = get_unaligned_le64(&sblk->fragment_table_start) /
-		ctxt.cur_dev->blksz;
+	start = get_unaligned_le64(&sblk->fragment_table_start) >>
+		ctxt.cur_dev->log2blksz;
 	n_blks = sqfs_calc_n_blks(sblk->fragment_table_start,
 				  sblk->export_table_start,
 				  &table_offset);
@@ -135,7 +135,7 @@ static int sqfs_frag_lookup(u32 inode_fragment_index,
 	start_block = get_unaligned_le64(table + table_offset + block *
 					 sizeof(u64));
 
-	start = start_block / ctxt.cur_dev->blksz;
+	start = start_block >> ctxt.cur_dev->log2blksz;
 	n_blks = sqfs_calc_n_blks(cpu_to_le64(start_block),
 				  sblk->fragment_table_start, &table_offset);
 
@@ -641,8 +641,8 @@ static int sqfs_read_inode_table(unsigned char **inode_table)
 
 	table_size = get_unaligned_le64(&sblk->directory_table_start) -
 		get_unaligned_le64(&sblk->inode_table_start);
-	start = get_unaligned_le64(&sblk->inode_table_start) /
-		ctxt.cur_dev->blksz;
+	start = get_unaligned_le64(&sblk->inode_table_start) >>
+		ctxt.cur_dev->log2blksz;
 	n_blks = sqfs_calc_n_blks(sblk->inode_table_start,
 				  sblk->directory_table_start, &table_offset);
 
@@ -725,8 +725,8 @@ static int sqfs_read_directory_table(unsigned char **dir_table, u32 **pos_list)
 	/* DIRECTORY TABLE */
 	table_size = get_unaligned_le64(&sblk->fragment_table_start) -
 		get_unaligned_le64(&sblk->directory_table_start);
-	start = get_unaligned_le64(&sblk->directory_table_start) /
-		ctxt.cur_dev->blksz;
+	start = get_unaligned_le64(&sblk->directory_table_start) >>
+		ctxt.cur_dev->log2blksz;
 	n_blks = sqfs_calc_n_blks(sblk->directory_table_start,
 				  sblk->fragment_table_start, &table_offset);
 
@@ -1334,11 +1334,11 @@ int sqfs_read(const char *filename, void *buf, loff_t offset, loff_t len,
 	}
 
 	for (j = 0; j < datablk_count; j++) {
-		start = data_offset / ctxt.cur_dev->blksz;
+		start = data_offset >> ctxt.cur_dev->log2blksz;
 		table_size = SQFS_BLOCK_SIZE(finfo.blk_sizes[j]);
 		table_offset = data_offset - (start * ctxt.cur_dev->blksz);
-		n_blks = DIV_ROUND_UP(table_size + table_offset,
-				      ctxt.cur_dev->blksz);
+		n_blks = (table_size + table_offset + ctxt.cur_dev->blksz - 1) >>
+			ctxt.cur_dev->log2blksz;
 
 		data_buffer = malloc_cache_aligned(n_blks * ctxt.cur_dev->blksz);
 
@@ -1388,10 +1388,10 @@ int sqfs_read(const char *filename, void *buf, loff_t offset, loff_t len,
 		goto free_buffer;
 	}
 
-	start = frag_entry.start / ctxt.cur_dev->blksz;
+	start = frag_entry.start >> ctxt.cur_dev->log2blksz;
 	table_size = SQFS_BLOCK_SIZE(frag_entry.size);
 	table_offset = frag_entry.start - (start * ctxt.cur_dev->blksz);
-	n_blks = DIV_ROUND_UP(table_size + table_offset, ctxt.cur_dev->blksz);
+	n_blks = (table_size + table_offset + ctxt.cur_dev->blksz - 1) >> ctxt.cur_dev->log2blksz;
 
 	fragment = malloc_cache_aligned(n_blks * ctxt.cur_dev->blksz);
 
fs/squashfs/sqfs_inode.c +1 -1
index 1368f3063c..5dbf945c80 100644
@@ -10,7 +10,7 @@
 #include <stdint.h>
 #include <stdio.h>
 #include <stdlib.h>
-#include <string.h>
+#include <part.h>
 
 #include "sqfs_decompressor.h"
 #include "sqfs_filesystem.h"

fs/squashfs/sqfs_inode.c int sqfs_inode_size(struct squashfs_base_inode *inode, u32 blk_size) · line 73

Quoted 8 lines
fs/squashfs/sqfs_inode.c +2 -2
@@ -68,9 +68,9 @@ int sqfs_inode_size(struct squashfs_base_inode *inode, u32 blk_size)
 		unsigned int blk_list_size;
 
 		if (fragment == 0xFFFFFFFF)
-			blk_list_size = DIV_ROUND_UP(file_size, blk_size);
+			blk_list_size = (file_size + blk_size - 1) >> LOG2(blk_size);
 		else
-			blk_list_size = file_size / blk_size;
+			blk_list_size = file_size > LOG2(blk_size);

one > is missing:

   blk_list_size = file_size >> LOG2(blk_size);
Quoted 3 lines
fs/squashfs/sqfs_inode.c +0 -0
 		return sizeof(*lreg) + blk_list_size * sizeof(u32);
 	}
Signature · 1 line
- Daniel
#3 lore raw

RFC PATCH v2 1/1 Fix missing __udivdi3 in SquashFS implementation.

In reply to Mauro Condarelli

Hello,

Quoted 7 lines · On Wed, 30 Sep 2020 17:45:11 +0200 Mauro Condarelli <[email protected]> wrote:

Use right shift to avoid 64-bit divisions.

These divisions are needed to convert from file length (potentially over 32-bit range) to block number, so result and remainder are guaranteed to fit in 32-bit integers.

Perhaps the commit log should contain an example U-Boot configuration/platform where it fails to build. Indeed, we did test the SquashFS code on ARM 32-bit, and it built and worked fine.

Quoted 17 lines

---

Changes in v2:
- replace division with right shift (Daniel Schwierzeck).
- remove vocore2-specific change (Daniel Schwierzeck).
- add warning to Kconfig about CONFIG_SYS_MALLOC_LEN (Tom Rini).

 fs/squashfs/Kconfig      |  2 ++
 fs/squashfs/sqfs.c       | 30 +++++++++++++++---------------
 fs/squashfs/sqfs_inode.c |  6 +++---
 3 files changed, 20 insertions(+), 18 deletions(-)
fs/squashfs/Kconfig +0 -0
index 54ab1618f1..7c3f83d007 100644

fs/squashfs/Kconfig config FS_SQUASHFS · line 13

Quoted 5 lines
fs/squashfs/Kconfig +2 -0
@@ -9,3 +9,5 @@ config FS_SQUASHFS
 	  filesystem use, for archival use (i.e. in cases where a .tar.gz file
 	  may be used), and in constrained block device/memory systems (e.g.
 	  embedded systems) where low overhead is needed.
+	  WARNING: if compression is enabled SquashFS needs a large amount
+	  of dynamic memory; make sure CONFIG_SYS_MALLOC_LEN >= 0x4000.

This change is completely unrelated, and should be in a separate patch.

fs/squashfs/Kconfig

Quoted 4 lines
fs/squashfs/Kconfig +2 -2
-		n_blks = DIV_ROUND_UP(table_size + table_offset,
-				      ctxt.cur_dev->blksz);
+		n_blks = (table_size + table_offset + ctxt.cur_dev->blksz - 1) >>
+			ctxt.cur_dev->log2blksz;

I understand why you have to add blksz - 1 before doing the shift, but I find that it's overall a lot less readable/clear. Is there a way to do better ?

We could use DIV_ROUND_UP_ULL() I guess.

fs/squashfs/Kconfig

Quoted 3 lines
fs/squashfs/Kconfig +1 -1
 		else
-			blk_list_size = file_size / blk_size;
+			blk_list_size = file_size > LOG2(blk_size);

Very bad mistake here: > should be >>.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

RFC PATCH v2 1/1 Fix missing __udivdi3 in SquashFS implementation.

Hello,

Thomas Petazzoni <[email protected]> wrote on Thu, 1 Oct 2020 09:28:41 +0200:

Quoted 29 lines

Hello,

Quoted 7 lines · On Wed, 30 Sep 2020 17:45:11 +0200 Mauro Condarelli <[email protected]> wrote:

Use right shift to avoid 64-bit divisions.

These divisions are needed to convert from file length (potentially over 32-bit range) to block number, so result and remainder are guaranteed to fit in 32-bit integers.

Perhaps the commit log should contain an example U-Boot configuration/platform where it fails to build. Indeed, we did test the SquashFS code on ARM 32-bit, and it built and worked fine.

Quoted 12 lines

---

Changes in v2:
- replace division with right shift (Daniel Schwierzeck).
- remove vocore2-specific change (Daniel Schwierzeck).
- add warning to Kconfig about CONFIG_SYS_MALLOC_LEN (Tom Rini).

 fs/squashfs/Kconfig      |  2 ++
 fs/squashfs/sqfs.c       | 30 +++++++++++++++---------------
 fs/squashfs/sqfs_inode.c |  6 +++---
 3 files changed, 20 insertions(+), 18 deletions(-)
Quoted 12 lines

fs/squashfs/Kconfig config FS_SQUASHFS · line 13

Quoted 10 lines
fs/squashfs/Kconfig +2 -0
index 54ab1618f1..7c3f83d007 100644
@@ -9,3 +9,5 @@ config FS_SQUASHFS
 	  filesystem use, for archival use (i.e. in cases where a .tar.gz file
 	  may be used), and in constrained block device/memory systems (e.g.
 	  embedded systems) where low overhead is needed.
+	  WARNING: if compression is enabled SquashFS needs a large amount
+	  of dynamic memory; make sure CONFIG_SYS_MALLOC_LEN >= 0x4000.  

This change is completely unrelated, and should be in a separate patch.

I was about to tell you the same thing, this warning is useful but should definitely lay into its own commit.

Quoted 9 lines

fs/squashfs/Kconfig

Quoted 4 lines
fs/squashfs/Kconfig +2 -2
-		n_blks = DIV_ROUND_UP(table_size + table_offset,
-				      ctxt.cur_dev->blksz);
+		n_blks = (table_size + table_offset + ctxt.cur_dev->blksz - 1) >>
+			ctxt.cur_dev->log2blksz;  

I understand why you have to add blksz - 1 before doing the shift, but I find that it's overall a lot less readable/clear. Is there a way to do better ?

Quoted 8 lines

We could use DIV_ROUND_UP_ULL() I guess.

fs/squashfs/Kconfig

Quoted 3 lines
fs/squashfs/Kconfig +1 -1
 		else
-			blk_list_size = file_size / blk_size;
+			blk_list_size = file_size > LOG2(blk_size);  

Very bad mistake here: > should be >>.

I personally highly dislike replacing divisions into shifts. I think it's too much effort when trying to understand code you did not write yourself. Is it possible to use something like do_div? plus, you can check the remainder to be 0 in this case.

Thanks,
Miqu?l

#5 lore raw

RFC PATCH v2 1/1 Fix missing __udivdi3 in SquashFS implementation.

Thanks for Your review.

Quoted 20 lines · Miquel Raynal

Hello,

Thomas Petazzoni <[email protected]> wrote on Thu, 1 Oct 2020 09:28:41 +0200:

Quoted 15 lines

Hello,

Quoted 7 lines · On Wed, 30 Sep 2020 17:45:11 +0200 Mauro Condarelli <[email protected]> wrote:

Use right shift to avoid 64-bit divisions.

These divisions are needed to convert from file length (potentially over 32-bit range) to block number, so result and remainder are guaranteed to fit in 32-bit integers.

Perhaps the commit log should contain an example U-Boot configuration/platform where it fails to build. Indeed, we did test the SquashFS code on ARM 32-bit, and it built and worked fine.

This fails on mips32, specifically for the vocore2 board. Problem here is __udivdi3 is not defined for this architecture while it is for ARM32.
My (limited) understanding suggests it should be removed for ARM since its usage has been (is being?) weeded out from both kernel and u-boot. This is not my call, though.

I will add a note to v3.

Quoted 24 lines
Quoted 13 lines
Quoted 13 lines

---

Changes in v2:
- replace division with right shift (Daniel Schwierzeck).
- remove vocore2-specific change (Daniel Schwierzeck).
- add warning to Kconfig about CONFIG_SYS_MALLOC_LEN (Tom Rini).

 fs/squashfs/Kconfig      |  2 ++
 fs/squashfs/sqfs.c       | 30 +++++++++++++++---------------
 fs/squashfs/sqfs_inode.c |  6 +++---
 3 files changed, 20 insertions(+), 18 deletions(-)
Quoted 11 lines

fs/squashfs/Kconfig config FS_SQUASHFS · line 13

Quoted 10 lines
fs/squashfs/Kconfig +2 -0
index 54ab1618f1..7c3f83d007 100644
@@ -9,3 +9,5 @@ config FS_SQUASHFS
 	  filesystem use, for archival use (i.e. in cases where a .tar.gz file
 	  may be used), and in constrained block device/memory systems (e.g.
 	  embedded systems) where low overhead is needed.
+	  WARNING: if compression is enabled SquashFS needs a large amount
+	  of dynamic memory; make sure CONFIG_SYS_MALLOC_LEN >= 0x4000.  

This change is completely unrelated, and should be in a separate patch.

Quoted 2 lines

I was about to tell you the same thing, this warning is useful but should definitely lay into its own commit.

Will do in v3.

Quoted 9 lines
Quoted 9 lines

fs/squashfs/Kconfig

Quoted 4 lines
fs/squashfs/Kconfig +2 -2
-		n_blks = DIV_ROUND_UP(table_size + table_offset,
-				      ctxt.cur_dev->blksz);
+		n_blks = (table_size + table_offset + ctxt.cur_dev->blksz - 1) >>
+			ctxt.cur_dev->log2blksz;  

I understand why you have to add blksz - 1 before doing the shift, but I find that it's overall a lot less readable/clear. Is there a way to do better ?

We could use DIV_ROUND_UP_ULL() I guess.

I did not do this because DIV_ROUND_UP() is in a global (non-architecture-specific) header and it unconditionally uses division; I did not know how to handle this. Would a comment suffice to clarify intent? Something like:

n_blks = (table_size + table_offset + ctxt.cur_dev->blksz - 1) >>

	  ctxt.cur_dev->log2blksz; /* ROUND_UP division */

Note: this problem stays even if we roll-back to use do_div(); see below.

Quoted 4 lines
Quoted 4 lines

fs/squashfs/Kconfig

Quoted 3 lines
fs/squashfs/Kconfig +1 -1
 		else
-			blk_list_size = file_size / blk_size;
+			blk_list_size = file_size > LOG2(blk_size);  

Very bad mistake here: > should be >>.

Sorry, my bad.
Corrected for v3.

Quoted 4 lines

I personally highly dislike replacing divisions into shifts. I think it's too much effort when trying to understand code you did not write yourself. Is it possible to use something like do_div? plus, you can check the remainder to be 0 in this case.

Please make up your mind about this.
I personally tend to agree with Miqu?l and my v1 patch used do_div() exclusively (i did not use even the lldiv() wrapper), but I will not insist either way... just let me know what's considered better.

Quoted 3 lines

Thanks,
Miqu?l

Many thanks
Mauro

#6 lore raw

RFC PATCH v2 1/1 Fix missing __udivdi3 in SquashFS implementation.

Correcting myself.
See below.

Quoted 69 lines · Mauro Condarelli

Thanks for Your review.

Quoted 20 lines · Miquel Raynal

Hello,

Thomas Petazzoni <[email protected]> wrote on Thu, 1 Oct 2020 09:28:41 +0200:

Quoted 15 lines

Hello,

Quoted 7 lines · On Wed, 30 Sep 2020 17:45:11 +0200 Mauro Condarelli <[email protected]> wrote:

Use right shift to avoid 64-bit divisions.

These divisions are needed to convert from file length (potentially over 32-bit range) to block number, so result and remainder are guaranteed to fit in 32-bit integers.

Perhaps the commit log should contain an example U-Boot configuration/platform where it fails to build. Indeed, we did test the SquashFS code on ARM 32-bit, and it built and worked fine.

This fails on mips32, specifically for the vocore2 board. Problem here is __udivdi3 is not defined for this architecture while it is for ARM32.
My (limited) understanding suggests it should be removed for ARM since its usage has been (is being?) weeded out from both kernel and u-boot. This is not my call, though.

I will add a note to v3.

Quoted 23 lines
Quoted 12 lines
Quoted 12 lines

---

Changes in v2:
- replace division with right shift (Daniel Schwierzeck).
- remove vocore2-specific change (Daniel Schwierzeck).
- add warning to Kconfig about CONFIG_SYS_MALLOC_LEN (Tom Rini).

 fs/squashfs/Kconfig      |  2 ++
 fs/squashfs/sqfs.c       | 30 +++++++++++++++---------------
 fs/squashfs/sqfs_inode.c |  6 +++---
 3 files changed, 20 insertions(+), 18 deletions(-)
Quoted 11 lines

fs/squashfs/Kconfig config FS_SQUASHFS · line 13

Quoted 10 lines
fs/squashfs/Kconfig +2 -0
index 54ab1618f1..7c3f83d007 100644
@@ -9,3 +9,5 @@ config FS_SQUASHFS
 	  filesystem use, for archival use (i.e. in cases where a .tar.gz file
 	  may be used), and in constrained block device/memory systems (e.g.
 	  embedded systems) where low overhead is needed.
+	  WARNING: if compression is enabled SquashFS needs a large amount
+	  of dynamic memory; make sure CONFIG_SYS_MALLOC_LEN >= 0x4000.  

This change is completely unrelated, and should be in a separate patch.

Quoted 2 lines

I was about to tell you the same thing, this warning is useful but should definitely lay into its own commit.

Will do in v3.

Quoted 9 lines
Quoted 9 lines

fs/squashfs/Kconfig

Quoted 4 lines
fs/squashfs/Kconfig +2 -2
-		n_blks = DIV_ROUND_UP(table_size + table_offset,
-				      ctxt.cur_dev->blksz);
+		n_blks = (table_size + table_offset + ctxt.cur_dev->blksz - 1) >>
+			ctxt.cur_dev->log2blksz;  

I understand why you have to add blksz - 1 before doing the shift, but I find that it's overall a lot less readable/clear. Is there a way to do better ?

We could use DIV_ROUND_UP_ULL() I guess.

Quoted 8 lines

I did not do this because DIV_ROUND_UP() is in a global (non-architecture-specific) header and it unconditionally uses division; I did not know how to handle this. Would a comment suffice to clarify intent? Something like:

n_blks = (table_size + table_offset + ctxt.cur_dev->blksz - 1) >>

	  ctxt.cur_dev->log2blksz; /* ROUND_UP division */

Note: this problem stays even if we roll-back to use do_div(); see below.

actually include/linux/kernel.h defines both DIV_ROUND_DOWN_ULL() and DIV_ROUND_UP_ULL() I suppose we should use those in all cases.

Quoted 11 lines
Quoted 4 lines
Quoted 4 lines

fs/squashfs/Kconfig

Quoted 3 lines
fs/squashfs/Kconfig +1 -1
 		else
-			blk_list_size = file_size / blk_size;
+			blk_list_size = file_size > LOG2(blk_size);  

Very bad mistake here: > should be >>.

Sorry, my bad.
Corrected for v3.

Quoted 4 lines

I personally highly dislike replacing divisions into shifts. I think it's too much effort when trying to understand code you did not write yourself. Is it possible to use something like do_div? plus, you can check the remainder to be 0 in this case.

Quoted 5 lines

Please make up your mind about this.
I personally tend to agree with Miqu?l and my v1 patch used do_div() exclusively (i did not use even the lldiv() wrapper), but I will not insist either way... just let me know what's considered better.

As said above: it seems using the macros is both "standard" and safer than using shifts.
If I get a go-ahead I'll use those macros in v3.

Quoted 5 lines
Quoted 3 lines

Thanks,
Miqu?l

Many thanks
Mauro

#7 lore raw

RFC PATCH v2 1/1 Fix missing __udivdi3 in SquashFS implementation.

Hi Mauro,

Mauro Condarelli <[email protected]> wrote on Thu, 1 Oct 2020 10:53:30 +0200:

Quoted 96 lines

Correcting myself.
See below.

Quoted 69 lines · Mauro Condarelli

Thanks for Your review.

Quoted 20 lines · Miquel Raynal

Hello,

Thomas Petazzoni <[email protected]> wrote on Thu, 1 Oct 2020 09:28:41 +0200:

Quoted 15 lines

Hello,

Quoted 7 lines · On Wed, 30 Sep 2020 17:45:11 +0200 Mauro Condarelli <[email protected]> wrote:

Use right shift to avoid 64-bit divisions.

These divisions are needed to convert from file length (potentially over 32-bit range) to block number, so result and remainder are guaranteed to fit in 32-bit integers.

Perhaps the commit log should contain an example U-Boot configuration/platform where it fails to build. Indeed, we did test the SquashFS code on ARM 32-bit, and it built and worked fine.

This fails on mips32, specifically for the vocore2 board. Problem here is __udivdi3 is not defined for this architecture while it is for ARM32.
My (limited) understanding suggests it should be removed for ARM since its usage has been (is being?) weeded out from both kernel and u-boot. This is not my call, though.

I will add a note to v3.

Quoted 23 lines
Quoted 12 lines
Quoted 12 lines

---

Changes in v2:
- replace division with right shift (Daniel Schwierzeck).
- remove vocore2-specific change (Daniel Schwierzeck).
- add warning to Kconfig about CONFIG_SYS_MALLOC_LEN (Tom Rini).

 fs/squashfs/Kconfig      |  2 ++
 fs/squashfs/sqfs.c       | 30 +++++++++++++++---------------
 fs/squashfs/sqfs_inode.c |  6 +++---
 3 files changed, 20 insertions(+), 18 deletions(-)
Quoted 11 lines

fs/squashfs/Kconfig config FS_SQUASHFS · line 13

Quoted 10 lines
fs/squashfs/Kconfig +2 -0
index 54ab1618f1..7c3f83d007 100644
@@ -9,3 +9,5 @@ config FS_SQUASHFS
 	  filesystem use, for archival use (i.e. in cases where a .tar.gz file
 	  may be used), and in constrained block device/memory systems (e.g.
 	  embedded systems) where low overhead is needed.
+	  WARNING: if compression is enabled SquashFS needs a large amount
+	  of dynamic memory; make sure CONFIG_SYS_MALLOC_LEN >= 0x4000.    

This change is completely unrelated, and should be in a separate patch.

Quoted 2 lines

I was about to tell you the same thing, this warning is useful but should definitely lay into its own commit.

Will do in v3.

Quoted 9 lines
Quoted 9 lines

fs/squashfs/Kconfig

Quoted 4 lines
fs/squashfs/Kconfig +2 -2
-		n_blks = DIV_ROUND_UP(table_size + table_offset,
-				      ctxt.cur_dev->blksz);
+		n_blks = (table_size + table_offset + ctxt.cur_dev->blksz - 1) >>
+			ctxt.cur_dev->log2blksz;    

I understand why you have to add blksz - 1 before doing the shift, but I find that it's overall a lot less readable/clear. Is there a way to do better ?

We could use DIV_ROUND_UP_ULL() I guess.

Quoted 8 lines

I did not do this because DIV_ROUND_UP() is in a global (non-architecture-specific) header and it unconditionally uses division; I did not know how to handle this. Would a comment suffice to clarify intent? Something like:

n_blks = (table_size + table_offset + ctxt.cur_dev->blksz - 1) >>

	  ctxt.cur_dev->log2blksz; /* ROUND_UP division */

Note: this problem stays even if we roll-back to use do_div(); see below.

actually include/linux/kernel.h defines both DIV_ROUND_DOWN_ULL() and DIV_ROUND_UP_ULL() I suppose we should use those in all cases.

Quoted 11 lines
Quoted 4 lines
Quoted 4 lines

fs/squashfs/Kconfig

Quoted 3 lines
fs/squashfs/Kconfig +1 -1
 		else
-			blk_list_size = file_size / blk_size;
+			blk_list_size = file_size > LOG2(blk_size);    

Very bad mistake here: > should be >>.

Sorry, my bad.
Corrected for v3.

Quoted 4 lines

I personally highly dislike replacing divisions into shifts. I think it's too much effort when trying to understand code you did not write yourself. Is it possible to use something like do_div? plus, you can check the remainder to be 0 in this case.

Quoted 8 lines
Quoted 5 lines

Please make up your mind about this.
I personally tend to agree with Miqu?l and my v1 patch used do_div() exclusively (i did not use even the lldiv() wrapper), but I will not insist either way... just let me know what's considered better.

As said above: it seems using the macros is both "standard" and safer than using shifts.
If I get a go-ahead I'll use those macros in v3.

I think we all agree using these macros is much nicer, readable and probably almost as fast.

Thanks,
Miqu?l

#8 lore raw

RFC PATCH v2 1/1 Fix missing __udivdi3 in SquashFS implementation.

Ok, Thanks.
Patch is ready, I'll send it after some extended testing on my system (vocore2).

Regards
Mauro

Quoted 115 lines · Miquel Raynal

Hi Mauro,

Mauro Condarelli <[email protected]> wrote on Thu, 1 Oct 2020 10:53:30 +0200:

Quoted 96 lines

Correcting myself.
See below.

Quoted 69 lines · Mauro Condarelli

Thanks for Your review.

Quoted 20 lines · Miquel Raynal

Hello,

Thomas Petazzoni <[email protected]> wrote on Thu, 1 Oct 2020 09:28:41 +0200:

Quoted 15 lines

Hello,

Quoted 7 lines · On Wed, 30 Sep 2020 17:45:11 +0200 Mauro Condarelli <[email protected]> wrote:

Use right shift to avoid 64-bit divisions.

These divisions are needed to convert from file length (potentially over 32-bit range) to block number, so result and remainder are guaranteed to fit in 32-bit integers.

Perhaps the commit log should contain an example U-Boot configuration/platform where it fails to build. Indeed, we did test the SquashFS code on ARM 32-bit, and it built and worked fine.

This fails on mips32, specifically for the vocore2 board. Problem here is __udivdi3 is not defined for this architecture while it is for ARM32.
My (limited) understanding suggests it should be removed for ARM since its usage has been (is being?) weeded out from both kernel and u-boot. This is not my call, though.

I will add a note to v3.

Quoted 23 lines
Quoted 12 lines
Quoted 12 lines

---

Changes in v2:
- replace division with right shift (Daniel Schwierzeck).
- remove vocore2-specific change (Daniel Schwierzeck).
- add warning to Kconfig about CONFIG_SYS_MALLOC_LEN (Tom Rini).

 fs/squashfs/Kconfig      |  2 ++
 fs/squashfs/sqfs.c       | 30 +++++++++++++++---------------
 fs/squashfs/sqfs_inode.c |  6 +++---
 3 files changed, 20 insertions(+), 18 deletions(-)
Quoted 11 lines

fs/squashfs/Kconfig config FS_SQUASHFS · line 13

Quoted 10 lines
fs/squashfs/Kconfig +2 -0
index 54ab1618f1..7c3f83d007 100644
@@ -9,3 +9,5 @@ config FS_SQUASHFS
 	  filesystem use, for archival use (i.e. in cases where a .tar.gz file
 	  may be used), and in constrained block device/memory systems (e.g.
 	  embedded systems) where low overhead is needed.
+	  WARNING: if compression is enabled SquashFS needs a large amount
+	  of dynamic memory; make sure CONFIG_SYS_MALLOC_LEN >= 0x4000.    

This change is completely unrelated, and should be in a separate patch.

Quoted 2 lines

I was about to tell you the same thing, this warning is useful but should definitely lay into its own commit.

Will do in v3.

Quoted 9 lines
Quoted 9 lines

fs/squashfs/Kconfig

Quoted 4 lines
fs/squashfs/Kconfig +2 -2
-		n_blks = DIV_ROUND_UP(table_size + table_offset,
-				      ctxt.cur_dev->blksz);
+		n_blks = (table_size + table_offset + ctxt.cur_dev->blksz - 1) >>
+			ctxt.cur_dev->log2blksz;    

I understand why you have to add blksz - 1 before doing the shift, but I find that it's overall a lot less readable/clear. Is there a way to do better ?

We could use DIV_ROUND_UP_ULL() I guess.

Quoted 8 lines

I did not do this because DIV_ROUND_UP() is in a global (non-architecture-specific) header and it unconditionally uses division; I did not know how to handle this. Would a comment suffice to clarify intent? Something like:

n_blks = (table_size + table_offset + ctxt.cur_dev->blksz - 1) >>

	  ctxt.cur_dev->log2blksz; /* ROUND_UP division */

Note: this problem stays even if we roll-back to use do_div(); see below.

actually include/linux/kernel.h defines both DIV_ROUND_DOWN_ULL() and DIV_ROUND_UP_ULL() I suppose we should use those in all cases.

Quoted 11 lines
Quoted 4 lines
Quoted 4 lines

fs/squashfs/Kconfig

Quoted 3 lines
fs/squashfs/Kconfig +1 -1
 		else
-			blk_list_size = file_size / blk_size;
+			blk_list_size = file_size > LOG2(blk_size);    

Very bad mistake here: > should be >>.

Sorry, my bad.
Corrected for v3.

Quoted 4 lines

I personally highly dislike replacing divisions into shifts. I think it's too much effort when trying to understand code you did not write yourself. Is it possible to use something like do_div? plus, you can check the remainder to be 0 in this case.

Quoted 8 lines
Quoted 5 lines

Please make up your mind about this.
I personally tend to agree with Miqu?l and my v1 patch used do_div() exclusively (i did not use even the lldiv() wrapper), but I will not insist either way... just let me know what's considered better.

As said above: it seems using the macros is both "standard" and safer than using shifts.
If I get a go-ahead I'll use those macros in v3.

I think we all agree using these macros is much nicer, readable and probably almost as fast.

Thanks,
Miqu?l