Skip to contentSkip to conversation

csum-file: add hashwrite_be64()

PATCH In review René Scharfe 3 participants 6 messages Nov 12, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 csum-file: add hashwrite_be64()

Add a helper function for hashing and writing 64-bit integers in network byte order. It returns the number of written bytes. This simplifies callers that keep track of the file offset, even though this number is a constant.

---
 csum-file.h | 7 +++++++
 1 file changed, 7 insertions(+)
csum-file.h +7 -0
index f9cbd317fb..e54d53d1d0 100644
@@ -62,4 +62,11 @@ static inline void hashwrite_be32(struct hashfile *f, uint32_t data)
 	hashwrite(f, &data, sizeof(data));
 }
 
+static inline size_t hashwrite_be64(struct hashfile *f, uint64_t data)
+{
+	data = htonll(data);
+	hashwrite(f, &data, sizeof(data));
+	return sizeof(data);
+}
+
 #endif

--
2.29.2

#1 lore raw

PATCH 2/3 midx: use hashwrite_be64()

Call hashwrite_be64() to write 64-bit values instead of open-coding it using hashwrite_be32() and sizeof. This shortens the code and makes its intent clearer.

---
And it saves me from headache induced by trying to remember whether "big endian" means that the most significant ("big") digit comes first (yes) or at the end, like the name suggests (no).

 midx.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
midx.c +2 -5
index d233b54ac7..da03c1449a 100644
@@ -785,9 +785,7 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off
 		if (!(offset >> 31))
 			continue;
 
-		hashwrite_be32(f, offset >> 32);
-		hashwrite_be32(f, offset & 0xffffffffUL);
-		written += 2 * sizeof(uint32_t);
+		written += hashwrite_be64(f, offset);
 
 		nr_large_offset--;
 	}
@@ -975,8 +973,7 @@ static int write_midx_internal(const char *object_dir, struct multi_pack_index *
 			    chunk_offsets[i]);
 
 		hashwrite_be32(f, chunk_ids[i]);
-		hashwrite_be32(f, chunk_offsets[i] >> 32);
-		hashwrite_be32(f, chunk_offsets[i]);
+		hashwrite_be64(f, chunk_offsets[i]);
 
 		written += MIDX_CHUNKLOOKUP_WIDTH;
 	}

--
2.29.2

#2 lore raw

PATCH 3/3 pack-write: use hashwrite_be64()

In reply to René Scharfe

Call hashwrite_be64() to write a 64-bit value instead of open-coding it using htonl() and hashwrite(). This shortens the code, gets rid of a buffer and several magic numbers, and makes the intent clearer.

---
 pack-write.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
pack-write.c +1 -4
index 23e19cc1ec..3513665e1e 100644
@@ -151,13 +151,10 @@ const char *write_idx_file(const char *index_name, struct pack_idx_entry **objec
 		while (nr_large_offset) {
 			struct pack_idx_entry *obj = *list++;
 			uint64_t offset = obj->offset;
-			uint32_t split[2];
 
 			if (!need_large_offset(offset, opts))
 				continue;
-			split[0] = htonl(offset >> 32);
-			split[1] = htonl(offset & 0xffffffff);
-			hashwrite(f, split, 8);
+			hashwrite_be64(f, offset);
 			nr_large_offset--;
 		}
 	}

--
2.29.2

#3 lore raw
In reply to René Scharfe
Quoted 7 lines · René Scharfe

Add a helper function for hashing and writing 64-bit integers in network byte order. It returns the number of written bytes. This simplifies callers that keep track of the file offset, even though this number is a constant.

These patches are absolutely correct, and I'm glad to see them show up in a very clear presentation. I had to go look to see why these were not already present, with [1] being the last instance of these showing up on-list. They did not get into the new version after a significant refactor [2].

Thanks,
-Stolee

[1] https://lore.kernel.org/git/[email protected]/
[2] https://lore.kernel.org/git/[email protected]/

#4 lore raw
Quoted 6 lines · Derrick Stolee
Quoted 5 lines · René Scharfe

Add a helper function for hashing and writing 64-bit integers in network byte order. It returns the number of written bytes. This simplifies callers that keep track of the file offset, even though this number is a constant.

Quoted 8 lines
Quoted 2 lines

These patches are absolutely correct, and I'm glad to see them show up in a very clear presentation. I had to go look to see why these were not already present, with [1] being the last instance of these showing up on-list. They did not get into the new version after a significant refactor [2].

That's right. What happened was we stopped writing the uncompressed "has an unpresentable Bloom filter" bitmap between the two versions you're talking about. Since we wrote each word as a big-endian 8-byte unsigned value, introducing hashwrite_be64() was useful in the earlier version.

But in the rewritten version, there were no _new_ callers that would have wanted hashwrite_be64(), so I dropped those patches since the series was already large.

That all being said, these are definitely useful patches to have, so I'm glad to see them being dug back up. Thanks, René :-).

Quoted 5 lines

Thanks,
-Stolee

[1] https://lore.kernel.org/git/[email protected]/
[2] https://lore.kernel.org/git/[email protected]/

Thanks,
Taylor

#5 lore raw
In reply to René Scharfe
Quoted 17 lines · René Scharfe

Add a helper function for hashing and writing 64-bit integers in network byte order. It returns the number of written bytes. This simplifies callers that keep track of the file offset, even though this number is a constant.

---
 csum-file.h | 7 +++++++
 1 file changed, 7 insertions(+)
csum-file.h +0 -0
index f9cbd317fb..e54d53d1d0 100644

csum-file.h static inline void hashwrite_be32(struct hashfile *f, uint32_t data) · line 67

Quoted 6 lines
csum-file.h +3 -0
@@ -62,4 +62,11 @@ static inline void hashwrite_be32(struct hashfile *f, uint32_t data)
 	hashwrite(f, &data, sizeof(data));
 }
 
+static inline size_t hashwrite_be64(struct hashfile *f, uint64_t data)
+{
+	data = htonll(data);

Great. This is new from my patch (which wrote the high- and low four bytes with two separate hashwrite_be32()'s), but I think it's a clear improvement.

In addition to being more readable, we can use the bswap instruction once instead of twice if it exists.

Please feel free to add my:

Thanks,
Taylor