read-cache: call verify_hdr() in a background thread

PATCH In review [email protected] 2 participants 7 messages Apr 25, 2017 – Apr 27, 2017

lore

Share & tools

lore

[email protected]

author

2017-04-25 18:41 UTC #0 lore raw

PATCH v8 1/1 read-cache: call verify_hdr() in a background thread

From: Jeff Hostetler <[email protected]>

Version 8 of this patch converts the unit test to use perl to corrupt the index checksum (rather than altering a filename) and also verifies the fsck error message as suggested in response to v7 on the mailing list.

If there are no other suggestions, I think this version should be considered final.

Jeff Hostetler (1):
read-cache: force_verify_index_checksum

 builtin/fsck.c  |  1 +
 cache.h         |  2 ++
 read-cache.c    |  7 +++++++
 t/t1450-fsck.sh | 32 ++++++++++++++++++++++++++++++++
 4 files changed, 42 insertions(+)

Signature · 1 line

2.9.3

[email protected]

author

2017-04-25 18:41 UTC #1 lore raw

PATCH v8 1/1 read-cache: force_verify_index_checksum

From: Jeff Hostetler <[email protected]>

Teach git to skip verification of the SHA1-1 checksum at the end of the index file in verify_hdr() which is called from read_index() unless the "force_verify_index_checksum" global variable is set.

Teach fsck to force this verification.

The checksum verification is for detecting disk corruption, and for small projects, the time it takes to compute SHA-1 is not that significant, but for gigantic repositories this calculation adds significant time to every command.

These effect can be seen using t/perf/p0002-read-cache.sh:

Test HEAD~1 HEAD
--------------------------------------------------------------------------------------
0002.1: read_cache/discard_cache 1000 times 0.66(0.44+0.20) 0.30(0.27+0.02) -54.5%

Signed-off-by Jeff Hostetler <[email protected]>

---
 builtin/fsck.c  |  1 +
 cache.h         |  2 ++
 read-cache.c    |  7 +++++++
 t/t1450-fsck.sh | 32 ++++++++++++++++++++++++++++++++
 4 files changed, 42 insertions(+)

builtin/fsck.c +1 -0

index 1a5cacc..5512d06 100644
@@ -771,6 +771,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 	}
 
 	if (keep_cache_objects) {
+		verify_index_checksum = 1;
 		read_cache();
 		for (i = 0; i < active_nr; i++) {
 			unsigned int mode;

cache.h +2 -0

index 80b6372..87f13bf 100644
@@ -685,6 +685,8 @@ extern void update_index_if_able(struct index_state *, struct lock_file *);
 extern int hold_locked_index(struct lock_file *, int);
 extern void set_alternate_index_output(const char *);
 
+extern int verify_index_checksum;
+
 /* Environment bits from configuration mechanism */
 extern int trust_executable_bit;
 extern int trust_ctime;

read-cache.c +7 -0

index 9054369..c4205aa 100644
@@ -1371,6 +1371,9 @@ struct ondisk_cache_entry_extended {
 			    ondisk_cache_entry_extended_size(ce_namelen(ce)) : \
 			    ondisk_cache_entry_size(ce_namelen(ce)))
 
+/* Allow fsck to force verification of the index checksum. */
+int verify_index_checksum;
+
 static int verify_hdr(struct cache_header *hdr, unsigned long size)
 {
 	git_SHA_CTX c;

@@ -1382,6 +1385,10 @@ static int verify_hdr(struct cache_header *hdr, unsigned long size)
 	hdr_version = ntohl(hdr->hdr_version);
 	if (hdr_version < INDEX_FORMAT_LB || INDEX_FORMAT_UB < hdr_version)
 		return error("bad index version %d", hdr_version);
+
+	if (!verify_index_checksum)
+		return 0;
+
 	git_SHA1_Init(&c);
 	git_SHA1_Update(&c, hdr, size - 20);
 	git_SHA1_Final(sha1, &c);

t/t1450-fsck.sh +32 -0

index 33a51c9..eff1cd6 100755
@@ -689,4 +689,36 @@ test_expect_success 'bogus head does not fallback to all heads' '
 	! grep $blob out
 '
 
+# Corrupt the checksum on the index.
+# Add 1 to the last byte in the SHA.
+corrupt_index_checksum () {
+    perl -w -e '
+	use Fcntl ":seek";
+	open my $fh, "+<", ".git/index" or die "open: $!";
+	binmode $fh;
+	seek $fh, -1, SEEK_END or die "seek: $!";
+	read $fh, my $in_byte, 1 or die "read: $!";
+
+	$in_value = unpack("C", $in_byte);
+	$out_value = ($in_value + 1) & 255;
+
+	$out_byte = pack("C", $out_value);
+
+	seek $fh, -1, SEEK_END or die "seek: $!";
+	print $fh $out_byte;
+	close $fh or die "close: $!";
+    '
+}
+
+# Corrupt the checksum on the index and then
+# verify that only fsck notices.
+test_expect_success 'detect corrupt index file in fsck' '
+	cp .git/index .git/index.backup &&
+	test_when_finished "mv .git/index.backup .git/index" &&
+	corrupt_index_checksum &&
+	test_must_fail git fsck --cache 2>expect &&
+	grep "bad index file" expect &&
+	git status
+'
+
 test_done

Signature · 1 line

2.9.3

Junio C Hamano

2017-04-26 04:11 UTC #2 lore raw

In reply to [email protected]

Quoted 9 lines · [email protected] writes:

From: Jeff Hostetler <[email protected]>
Version 8 of this patch converts the unit test to use perl to corrupt the index checksum (rather than altering a filename) and also verifies the fsck error message as suggested in response to v7 on the mailing list.
If there are no other suggestions, I think this version should be considered final.

Oops. The earlier one has already been in 'master' for a few days. Let's make this an incremental update.

Is the description in the following something you are OK with (so that I can add your sign-off)?

Thanks.

-- >8 --
From: Jeff Hostetler <[email protected]>
Date: Tue, 25 Apr 2017 18:41:09 +0000
Subject: t1450: avoid use of "sed" on the index, which is a binary file

The previous step added a path zzzzzzzz to the index, and then used "sed" to replace this string to yyyyyyyy to create a test case where the checksum at the end of the file does not match the contents.

Unfortunately, use of "sed" on a non-text file is not portable. Instead, use a Perl script that seeks to the end and modifies the last byte of the file (where we _know_ stores the trailing checksum).

---
 t/t1450-fsck.sh | 33 ++++++++++++++++++++++++++-------
 1 file changed, 26 insertions(+), 7 deletions(-)

t/t1450-fsck.sh +26 -7

index 677e15a7a4..eff1cd68e9 100755
@@ -689,16 +689,35 @@ test_expect_success 'bogus head does not fallback to all heads' '
 	! grep $blob out
 '
 
+# Corrupt the checksum on the index.
+# Add 1 to the last byte in the SHA.
+corrupt_index_checksum () {
+    perl -w -e '
+	use Fcntl ":seek";
+	open my $fh, "+<", ".git/index" or die "open: $!";
+	binmode $fh;
+	seek $fh, -1, SEEK_END or die "seek: $!";
+	read $fh, my $in_byte, 1 or die "read: $!";
+
+	$in_value = unpack("C", $in_byte);
+	$out_value = ($in_value + 1) & 255;
+
+	$out_byte = pack("C", $out_value);
+
+	seek $fh, -1, SEEK_END or die "seek: $!";
+	print $fh $out_byte;
+	close $fh or die "close: $!";
+    '
+}
+
+# Corrupt the checksum on the index and then
+# verify that only fsck notices.
 test_expect_success 'detect corrupt index file in fsck' '
 	cp .git/index .git/index.backup &&
 	test_when_finished "mv .git/index.backup .git/index" &&
-	echo zzzzzzzz >zzzzzzzz &&
-	git add zzzzzzzz &&
-	sed -e "s/zzzzzzzz/yyyyyyyy/" .git/index >.git/index.yyy &&
-	mv .git/index.yyy .git/index &&
-	# Confirm that fsck detects invalid checksum
-	test_must_fail git fsck --cache &&
-	# Confirm that status no longer complains about invalid checksum
+	corrupt_index_checksum &&
+	test_must_fail git fsck --cache 2>expect &&
+	grep "bad index file" expect &&
 	git status
 '

Junio C Hamano

2017-04-26 04:34 UTC #3 lore raw

Quoted 12 lines · Junio C Hamano

Quoted 9 lines · [email protected] writes:
From: Jeff Hostetler <[email protected]>
Version 8 of this patch converts the unit test to use perl to corrupt the index checksum (rather than altering a filename) and also verifies the fsck error message as suggested in response to v7 on the mailing list.
If there are no other suggestions, I think this version should be considered final.

Quoted 7 lines

Oops. The earlier one has already been in 'master' for a few days. Let's make this an incremental update.
Is the description in the following something you are OK with (so that I can add your sign-off)?
Thanks.

By the way, I said I am fine with the two-liner version in Dscho's message, but I am also OK with this version that does not use a separate dd and instead does everything in a single invocation of Perl. As long as you've tested this version, there is no point replacing it with yet another one.

Thanks.

Quoted 66 lines

-- >8 --
From: Jeff Hostetler <[email protected]>
Date: Tue, 25 Apr 2017 18:41:09 +0000
Subject: t1450: avoid use of "sed" on the index, which is a binary file

Unfortunately, use of "sed" on a non-text file is not portable. Instead, use a Perl script that seeks to the end and modifies the last byte of the file (where we _know_ stores the trailing checksum).

---
 t/t1450-fsck.sh | 33 ++++++++++++++++++++++++++-------
 1 file changed, 26 insertions(+), 7 deletions(-)

t/t1450-fsck.sh +26 -7

index 677e15a7a4..eff1cd68e9 100755
@@ -689,16 +689,35 @@ test_expect_success 'bogus head does not fallback to all heads' '
 	! grep $blob out
 '
 
+# Corrupt the checksum on the index.
+# Add 1 to the last byte in the SHA.
+corrupt_index_checksum () {
+    perl -w -e '
+	use Fcntl ":seek";
+	open my $fh, "+<", ".git/index" or die "open: $!";
+	binmode $fh;
+	seek $fh, -1, SEEK_END or die "seek: $!";
+	read $fh, my $in_byte, 1 or die "read: $!";
+
+	$in_value = unpack("C", $in_byte);
+	$out_value = ($in_value + 1) & 255;
+
+	$out_byte = pack("C", $out_value);
+
+	seek $fh, -1, SEEK_END or die "seek: $!";
+	print $fh $out_byte;
+	close $fh or die "close: $!";
+    '
+}
+
+# Corrupt the checksum on the index and then
+# verify that only fsck notices.
 test_expect_success 'detect corrupt index file in fsck' '
 	cp .git/index .git/index.backup &&
 	test_when_finished "mv .git/index.backup .git/index" &&
-	echo zzzzzzzz >zzzzzzzz &&
-	git add zzzzzzzz &&
-	sed -e "s/zzzzzzzz/yyyyyyyy/" .git/index >.git/index.yyy &&
-	mv .git/index.yyy .git/index &&
-	# Confirm that fsck detects invalid checksum
-	test_must_fail git fsck --cache &&
-	# Confirm that status no longer complains about invalid checksum
+	corrupt_index_checksum &&
+	test_must_fail git fsck --cache 2>expect &&
+	grep "bad index file" expect &&
 	git status
 '

Jeff Hostetler

author

2017-04-26 13:06 UTC #4 lore raw

Quoted 19 lines · Junio C Hamano

Quoted 17 lines · Junio C Hamano
Quoted 9 lines · [email protected] writes:
From: Jeff Hostetler <[email protected]>
Version 8 of this patch converts the unit test to use perl to corrupt the index checksum (rather than altering a filename) and also verifies the fsck error message as suggested in response to v7 on the mailing list.
If there are no other suggestions, I think this version should be considered final.
Oops. The earlier one has already been in 'master' for a few days. Let's make this an incremental update.
Is the description in the following something you are OK with (so that I can add your sign-off)?

Quoted 6 lines

Quoted 1 line
Thanks.
By the way, I said I am fine with the two-liner version in Dscho's message, but I am also OK with this version that does not use a separate dd and instead does everything in a single invocation of Perl. As long as you've tested this version, there is no point replacing it with yet another one.

Either version is fine. I'd say use my perl version as I have tested it and it is simple enough and already in the tree. I don't think it's worth the bother to switch it to the dd version.

Thanks
Jeff

Quoted 68 lines

Thanks.

Quoted 66 lines

-- >8 --
From: Jeff Hostetler <[email protected]>
Date: Tue, 25 Apr 2017 18:41:09 +0000
Subject: t1450: avoid use of "sed" on the index, which is a binary file

Unfortunately, use of "sed" on a non-text file is not portable. Instead, use a Perl script that seeks to the end and modifies the last byte of the file (where we _know_ stores the trailing checksum).

---
  t/t1450-fsck.sh | 33 ++++++++++++++++++++++++++-------
  1 file changed, 26 insertions(+), 7 deletions(-)

t/t1450-fsck.sh +26 -7

index 677e15a7a4..eff1cd68e9 100755
@@ -689,16 +689,35 @@ test_expect_success 'bogus head does not fallback to all heads' '
  	! grep $blob out
  '
 
+# Corrupt the checksum on the index.
+# Add 1 to the last byte in the SHA.
+corrupt_index_checksum () {
+    perl -w -e '
+	use Fcntl ":seek";
+	open my $fh, "+<", ".git/index" or die "open: $!";
+	binmode $fh;
+	seek $fh, -1, SEEK_END or die "seek: $!";
+	read $fh, my $in_byte, 1 or die "read: $!";
+
+	$in_value = unpack("C", $in_byte);
+	$out_value = ($in_value + 1) & 255;
+
+	$out_byte = pack("C", $out_value);
+
+	seek $fh, -1, SEEK_END or die "seek: $!";
+	print $fh $out_byte;
+	close $fh or die "close: $!";
+    '
+}
+
+# Corrupt the checksum on the index and then
+# verify that only fsck notices.
  test_expect_success 'detect corrupt index file in fsck' '
  	cp .git/index .git/index.backup &&
  	test_when_finished "mv .git/index.backup .git/index" &&
-	echo zzzzzzzz >zzzzzzzz &&
-	git add zzzzzzzz &&
-	sed -e "s/zzzzzzzz/yyyyyyyy/" .git/index >.git/index.yyy &&
-	mv .git/index.yyy .git/index &&
-	# Confirm that fsck detects invalid checksum
-	test_must_fail git fsck --cache &&
-	# Confirm that status no longer complains about invalid checksum
+	corrupt_index_checksum &&
+	test_must_fail git fsck --cache 2>expect &&
+	grep "bad index file" expect &&
  	git status
  '

Junio C Hamano

2017-04-27 00:41 UTC #5 lore raw

Quoted 3 lines · Jeff Hostetler

Either version is fine. I'd say use my perl version as I have tested it and it is simple enough and already in the tree. I don't think it's worth the bother to switch it to the dd version.

Thanks, I agree what you said is very sensible.

Jeff Hostetler

author

2017-04-26 13:03 UTC #6 lore raw

In reply to Junio C Hamano

Quoted 9 lines · Junio C Hamano

Quoted 7 lines · [email protected] writes:
From: Jeff Hostetler <[email protected]>
Version 8 of this patch converts the unit test to use perl to corrupt the index checksum (rather than altering a filename) and also verifies the fsck error message as suggested in response to v7 on the mailing list.

Quoted 7 lines

Quoted 2 lines
If there are no other suggestions, I think this version should be considered final.
Oops. The earlier one has already been in 'master' for a few days. Let's make this an incremental update.
Is the description in the following something you are OK with (so that I can add your sign-off)?

Yes, this is fine. Thanks!
Jeff

Quoted 69 lines

Thanks.

-- >8 --
From: Jeff Hostetler <[email protected]>
Date: Tue, 25 Apr 2017 18:41:09 +0000
Subject: t1450: avoid use of "sed" on the index, which is a binary file

Unfortunately, use of "sed" on a non-text file is not portable. Instead, use a Perl script that seeks to the end and modifies the last byte of the file (where we _know_ stores the trailing checksum).

---
  t/t1450-fsck.sh | 33 ++++++++++++++++++++++++++-------
  1 file changed, 26 insertions(+), 7 deletions(-)

t/t1450-fsck.sh +26 -7

index 677e15a7a4..eff1cd68e9 100755
@@ -689,16 +689,35 @@ test_expect_success 'bogus head does not fallback to all heads' '
  	! grep $blob out
  '
 
+# Corrupt the checksum on the index.
+# Add 1 to the last byte in the SHA.
+corrupt_index_checksum () {
+    perl -w -e '
+	use Fcntl ":seek";
+	open my $fh, "+<", ".git/index" or die "open: $!";
+	binmode $fh;
+	seek $fh, -1, SEEK_END or die "seek: $!";
+	read $fh, my $in_byte, 1 or die "read: $!";
+
+	$in_value = unpack("C", $in_byte);
+	$out_value = ($in_value + 1) & 255;
+
+	$out_byte = pack("C", $out_value);
+
+	seek $fh, -1, SEEK_END or die "seek: $!";
+	print $fh $out_byte;
+	close $fh or die "close: $!";
+    '
+}
+
+# Corrupt the checksum on the index and then
+# verify that only fsck notices.
  test_expect_success 'detect corrupt index file in fsck' '
  	cp .git/index .git/index.backup &&
  	test_when_finished "mv .git/index.backup .git/index" &&
-	echo zzzzzzzz >zzzzzzzz &&
-	git add zzzzzzzz &&
-	sed -e "s/zzzzzzzz/yyyyyyyy/" .git/index >.git/index.yyy &&
-	mv .git/index.yyy .git/index &&
-	# Confirm that fsck detects invalid checksum
-	test_must_fail git fsck --cache &&
-	# Confirm that status no longer complains about invalid checksum
+	corrupt_index_checksum &&
+	test_must_fail git fsck --cache 2>expect &&
+	grep "bad index file" expect &&
  	git status
  '