Bug in 2.48 with `git refs migrate`
Hi,
I noticed that Git 2.48 has support for migrating refs when there are reflogs and, as promised at Git Merge, I decided to try it out. Unfortunately, I got an error:
----
% git refs migrate --ref-format=reftable --dry-run
error: reftable: transaction failure: corrupt reftable file
----
Here's a small reproduction case:
----
#!/bin/sh
rm -fr test-repo
git init -b dev test-repo
cd test-repo
# start first block touch foo.txt git add foo.txt git commit -m +
head=$(git rev-parse HEAD)
seq 5000 | sed -Ee "s!^(.*)\$!create refs/heads/ref-\1 $head!" | git update-ref --stdin
# end first block
# start second block echo abc >bar.txt git add bar.txt git commit -m + head=$(git rev-parse HEAD) seq 3000 | sed -Ee "s!^(.*)\$!update refs/heads/ref-\1 $head!" | git update-ref --stdin # end second block
git refs migrate --ref-format=reftable
----
I can also reproduce this on the latest master.
If you remove the second block, it does not appear to reproduce. Some investigation led me to the conclusion that the difference is when max_update_index is not 1, the header has the value 1 for it but the trailer has the correct value, and so we flag the header and trailer as mismatching and therefore it gets marked as corrupt. I believe the reason things work when removing the second block is because that value remains 1, and so it works.
I haven't done anything else to investigate here, for which I apologize, but I just wanted to mention it while it was fresh on my mind.
In case this is helpful, I did see this when attempting to migrate two work repositories with lots of reflogs and many refs (the smaller has 2983 and the larger, 44832). I obviously cannot send you these repositories or things in them, but I'm happy to test patches against them.
Please let me know if I can provide more useful information.
Signature · 2 lines
brian m. carlson (they/them or he/him) Toronto, Ontario, CA
Quoted 7 lines · brian m. carlson
If you remove the second block, it does not appear to reproduce. Some investigation led me to the conclusion that the difference is when
max_update_indexis not 1, the header has the value 1 for it but the trailer has the correct value, and so we flag the header and trailer as mismatching and therefore it gets marked as corrupt. I believe the reason things work when removing the second block is because that value remains 1, and so it works.
Hm. Makes me wonder whether this is caused by the newly added code to also migrate reflog entries, as we'd play around with the update index only when migrating those. Cc'ing Karthik so he can have a look.
Quoted 10 lines
I haven't done anything else to investigate here, for which I apologize, but I just wanted to mention it while it was fresh on my mind.
In case this is helpful, I did see this when attempting to migrate two work repositories with lots of reflogs and many refs (the smaller has 2983 and the larger, 44832). I obviously cannot send you these repositories or things in them, but I'm happy to test patches against them.
Please let me know if I can provide more useful information.
Thanks for your report!
Patrick
"brian m. carlson" <[email protected]> writes:
Hello brian,
Quoted 6 lines
Hi,
I noticed that Git 2.48 has support for migrating refs when there are reflogs and, as promised at Git Merge, I decided to try it out. Unfortunately, I got an error:
Thanks for trying it out and reporting.
Quoted 32 lines
----
% git refs migrate --ref-format=reftable --dry-run
error: reftable: transaction failure: corrupt reftable file
----Here's a small reproduction case:
----
#!/bin/shrm -fr test-repo
git init -b dev test-repo
cd test-repo# start first block touch foo.txt git add foo.txt git commit -m +head=$(git rev-parse HEAD)
seq 5000 | sed -Ee "s!^(.*)\$!create refs/heads/ref-\1 $head!" | git update-ref --stdin# end first block# start second block echo abc >bar.txt git add bar.txt git commit -m + head=$(git rev-parse HEAD) seq 3000 | sed -Ee "s!^(.*)\$!update refs/heads/ref-\1 $head!" | git update-ref --stdin # end second block
Quoted 5 lines
git refs migrate --ref-format=reftable
----I can also reproduce this on the latest master.
I can also reproduce it using the script provided here.
Quoted 9 lines
If you remove the second block, it does not appear to reproduce. Some investigation led me to the conclusion that the difference is when
max_update_indexis not 1, the header has the value 1 for it but the trailer has the correct value, and so we flag the header and trailer as mismatching and therefore it gets marked as corrupt. I believe the reason things work when removing the second block is because that value remains 1, and so it works.
Indeed, you're absolutely correct here, this is because the max_update_index is indeed different between the footer and the header. This flow is triggered when we have multiple reftable blocks being written with multiple reflog entries. The multiple blocks is crucial because we update the max_update_index right before writing the reflogs. So if there are refs being written and they spawn more than one block, the header is written without the updated max_update_index, causing the mismatch.
Quoted 3 lines
I haven't done anything else to investigate here, for which I apologize, but I just wanted to mention it while it was fresh on my mind.
I think you've provided sufficient information and also enough to make it easy for me to debug.
Quoted 6 lines
In case this is helpful, I did see this when attempting to migrate two work repositories with lots of reflogs and many refs (the smaller has 2983 and the larger, 44832). I obviously cannot send you these repositories or things in them, but I'm happy to test patches against them.
I'm attaching a patch below which should fixes the issue for me and also adding a test to test against the same. I'd be grateful if you could also test the patch against the repositoryies you mention.
Quoted 4 lines
Please let me know if I can provide more useful information. --
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA
-- >8 --
Subject: [PATCH] reftable: write correct max_update_index to header
In 297c09eabb (refs: allow multiple reflog entries for the same refname, 2024-12-16), the reftable backend learned to handle multiple reflog entries within the same transaction. This was done modifying the update_index for reflogs with multiple indices. During writing the logs, the max_update_index of the writer was modified to ensure the limits were raised to the modified update_indexs.
However, since ref entries are written before the modification to the max_update_index, if there are multiple blocks to be written, the reftable backend writes the header with the old max_update_index. When all logs are finally written, the footer will be written with the new min_update_index. This causes a mismatch between the header and the footer and causes the reftable file to be corrupted. The existing tests only spawn a single block and since headers are lazily written with the first block, the tests didn't capture this bug.
To fix the issue, the appropriate max_update_index limit must be set even before the first block is written. Add a max_index field to the transaction which holds the max_index within all its updates, then propagate this value to the reftable backend, wherein this is used to the set the max_update_index correctly.
Add a test which creates a few thousand reference updates with multiple reflog entries, which should trigger the bug.
--- refs.c | 7 +++++++ refs/refs-internal.h | 1 + refs/reftable-backend.c | 20 ++++++++++---------- t/t1460-refs-migrate.sh | 12 ++++++++++++ 4 files changed, 30 insertions(+), 10 deletions(-)
refs.c +0 -0
@@ -1345,6 +1345,13 @@ int ref_transaction_update_reflog(struct
ref_transaction *transaction,
update->flags &= ~REF_HAVE_OLD;
update->index = index;
+ /*
+ * Reference backends may need to know the max index to optimize
+ * their writes. So we store the max_index on the transaction level.
+ */
+ if (index > transaction->max_index)
+ transaction->max_index = index;
+
return 0;
}
refs/refs-internal.h +1 -0
@@ -203,6 +203,7 @@ struct ref_transaction {enum ref_transaction_state state;void *backend_data;unsigned int flags; +unsigned int max_index; };/*
refs/reftable-backend.c +1 -0
@@ -942,6 +942,7 @@ struct write_transaction_table_arg {size_t updates_nr;size_t updates_alloc;size_t updates_expected; +unsigned int max_index; };struct reftable_transaction_data {
@@ -1428,7 +1429,6 @@ static int write_transaction_table(struct
reftable_writer *writer, void *cb_data
struct reftable_log_record *logs = NULL;
struct ident_split committer_ident = {0};
size_t logs_nr = 0, logs_alloc = 0, i;
- uint64_t max_update_index = ts;
const char *committer_info;
int ret = 0;
@@ -1438,7 +1438,12 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data
QSORT(arg->updates, arg->updates_nr, transaction_update_cmp);
- reftable_writer_set_limits(writer, ts, ts);
+ /*
+ * During reflog migration, we add indexes for a single reflog with
+ * multiple entries. Each entry will contain a different update_index,
+ * so set the limits accordingly.
+ */
+ reftable_writer_set_limits(writer, ts, ts + arg->max_index);
for (i = 0; i < arg->updates_nr; i++) {
struct reftable_transaction_update *tx_update = &arg->updates[i];
@@ -1540,12 +1545,6 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data
*/
log->update_index = ts + u->index;
- /*
- * Note the max update_index so the limit can be set later on.
- */
- if (log->update_index > max_update_index)
- max_update_index = log->update_index;
-
log->refname = xstrdup(u->refname); memcpy(log->value.update.new_hash,
u->new_oid.hash, GIT_MAX_RAWSZ);
@@ -1609,8 +1608,6 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data
* and log blocks.
*/
if (logs) {
- reftable_writer_set_limits(writer, ts, max_update_index);
-
ret = reftable_writer_add_logs(writer, logs, logs_nr); if (ret < 0)
goto done;
@@ -1631,6 +1628,9 @@ static int reftable_be_transaction_finish(struct ref_store *ref_store UNUSED,
struct reftable_transaction_data *tx_data = transaction->backend_data; int ret = 0;
+ if (tx_data->args)
+ tx_data->args->max_index = transaction->max_index;
+
for (size_t i = 0; i < tx_data->args_nr; i++) {
ret = reftable_addition_add(tx_data->args[i].addition,
write_transaction_table, &tx_data->args[i]);
t/t1460-refs-migrate.sh +12 -0
@@ -227,6 +227,18 @@ do done done +test_expect_success 'multiple reftable blocks with multiple entries' ' + test_when_finished "rm -rf repo" && + git init --ref-format=files repo && + test_commit -C repo first && + printf "create refs/heads/ref-%d HEAD\n" $(test_seq 5000) >stdin && + git -C repo update-ref --stdin <stdin && + test_commit -C repo second && + printf "update refs/heads/ref-%d HEAD\n" $(test_seq 3000) >stdin && + git -C repo update-ref --stdin <stdin && + test_migration repo reftable +' + test_expect_success 'migrating from files format deletes backend files' ' test_when_finished "rm -rf repo" && git init --ref-format=files repo &&
Signature · 1 line
2.47.0
Quoted 5 lines · Karthik Nayak
Subject: [PATCH] reftable: write correct
max_update_indexto headerIn
297c09eabb(refs: allow multiple reflog entries for the same refname, 2024-12-16), the reftable backend learned to handle multiple reflog entries within the same transaction. This was done modifying the
"done" -> "done by", I think.
refs.c int ref_transaction_update_reflog(struct
Quoted 12 lines
To fix the issue, the appropriate
max_update_indexlimit must be set even before the first block is written. Add amax_indexfield to the transaction which holds themax_indexwithin all its updates, then propagate this value to the reftable backend, wherein this is used to the set themax_update_indexcorrectly.refs.c +0
-0@@ -1345,6 +1345,13 @@ int ref_transaction_update_reflog(struct
Not the focus of this topic/fix, but I notice that the only caller of this ref_transaction_update_reflog() function is a static function migrate_one_reflog_entry() in the same file. Do we expect that we would add more callers? Otherwise we should make it a file scope static and remove it from <refs.h> file.
refs.c int ref_transaction_update_reflog(struct · line 1357
Quoted 13 lines
refs.c +7
-0ref_transaction *transaction, update->flags &= ~REF_HAVE_OLD ; update->index = index; +/* +* Reference backends may need to know the max index to optimize +* their writes. So we store the max_index on the transaction level. +*/ +if (index > transaction->max_index) + transaction->max_index = index; +return 0 ; }
So from the problem description, whenever we consume an index by assigning it to an update that belongs to a transaction, we must make sure that transaction's max_index covers that value of the index? I was wondering if we should have a less error prone way to do that by having a helper function that takes ref_update and ref_transaction objects to do the above, but this is exclusively used by reflog migration code path and nowhere else, so it may probably be fine as-is. I dunno.
Quoted 10 lines · Junio C Hamano
Quoted 5 lines · Karthik Nayak
Subject: [PATCH] reftable: write correct
max_update_indexto headerIn
297c09eabb(refs: allow multiple reflog entries for the same refname, 2024-12-16), the reftable backend learned to handle multiple reflog entries within the same transaction. This was done modifying the"done" -> "done by", I think.
Yup, thanks!
Quoted 7 lines
Quoted 5 lines
To fix the issue, the appropriate
max_update_indexlimit must be set even before the first block is written. Add amax_indexfield to the transaction which holds themax_indexwithin all its updates, then propagate this value to the reftable backend, wherein this is used to the set themax_update_indexcorrectly.
Quoted 12 lines
refs.c int ref_transaction_update_reflog(struct
Quoted 5 lines
refs.c +0
-0@@ -1345,6 +1345,13 @@ int ref_transaction_update_reflog(structNot the focus of this topic/fix, but I notice that the only caller of this
ref_transaction_update_reflog()function is a static functionmigrate_one_reflog_entry()in the same file. Do we expect that we would add more callers? Otherwise we should make it a file scope static and remove it from <refs.h> file.
I don't think we expect any other callers now. Maybe someday we'd expose creating reflogs to the users. But for now, making it static is more worthwhile.
Let's follow the boy scout rule and clean this up, I'll add a commit to do the same in v2 of my patch.
Quoted 14 lines
refs.c int ref_transaction_update_reflog(struct · line 1357
Quoted 13 lines
refs.c +7
-0ref_transaction *transaction, update->flags &= ~REF_HAVE_OLD ; update->index = index; +/* +* Reference backends may need to know the max index to optimize +* their writes. So we store the max_index on the transaction level. +*/ +if (index > transaction->max_index) + transaction->max_index = index; +return 0 ; }
Quoted 8 lines
So from the problem description, whenever we consume an index by assigning it to an update that belongs to a transaction, we must make sure that transaction's
max_indexcovers that value of the index? I was wondering if we should have a less error prone way to do that by having a helper function that takesref_updateandref_transactionobjects to do the above, but this is exclusively used by reflog migration code path and nowhere else, so it may probably be fine as-is. I dunno.
Yup, that's correct. It should be okay, but let me do it anyways, makes it safer for the future.
Quoted 5 lines · Karthik Nayak
"brian m. carlson" <[email protected]> writes:
I'm attaching a patch below which should fixes the issue for me and also adding a test to test against the same. I'd be grateful if you could also test the patch against the repositoryies you mention.
Fantastic, I'll try to do that tomorrow and get back to you. I really appreciate such a prompt response.
Quoted 17 lines
refs/refs-internal.h +1
-0@@ -203,6 +203,7 @@ struct ref_transaction {enum ref_transaction_state state;void *backend_data;unsigned int flags; +unsigned int max_index; };/* refs/reftable-backend.c +0
-0
refs/reftable-backend.c struct write_transaction_table_arg { · line 945
Quoted 4 lines
refs/reftable-backend.c +1
-0@@ -942,6 +942,7 @@ struct write_transaction_table_arg {size_t updates_nr;size_t updates_alloc;size_t updates_expected; +unsigned int max_index;
I wonder if this and the above should be uint64_t instead of unsigned int. From the file names and the data format, it looks like we intentionally use a 64-bit integer. That's good, because I have unfortunately seen some people who have created giant test repositories with really unreasonable numbers of commits and I could see us possibly exceeding a 32-bit integer here.
Signature · 2 lines
brian m. carlson (they/them or he/him) Toronto, Ontario, CA
Quoted 30 lines · brian m. carlson
Quoted 5 lines · Karthik Nayak
"brian m. carlson" <[email protected]> writes:
I'm attaching a patch below which should fixes the issue for me and also adding a test to test against the same. I'd be grateful if you could also test the patch against the repositoryies you mention.
Fantastic, I'll try to do that tomorrow and get back to you. I really appreciate such a prompt response.
Quoted 20 lines
refs/refs-internal.h +1
-0@@ -203,6 +203,7 @@ struct ref_transaction {enum ref_transaction_state state;void *backend_data;unsigned int flags; +unsigned int max_index; };/* refs/reftable-backend.c +0
-0@@ -942,6 +942,7 @@ struct write_transaction_table_arg {size_t updates_nr;size_t updates_alloc;size_t updates_expected;
Quoted 8 lines
refs/reftable-backend.c struct write_transaction_table_arg { · line 945
Quoted 1 line
refs/reftable-backend.c +1
-0+unsigned int max_index;I wonder if this and the above should be
uint64_tinstead ofunsigned int. From the file names and the data format, it looks like we intentionally use a 64-bit integer. That's good, because I have unfortunately seen some people who have created giant test repositories with really unreasonable numbers of commits and I could see us possibly exceeding a 32-bit integer here.
Ideally it should be okay, since this only comes into play when we're migrating reflogs. So the index would be the number of reflog entries for a given ref. I suppose even for large repositories this number should be low. But I'd rather be safe here, so let me modify this in the next version to be 'uint64_t'.
Quoted 3 lines
--
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA
Quoted 4 lines · Karthik Nayak
"brian m. carlson" <[email protected]> writes:
I'm attaching a patch below which should fixes the issue for me and also adding a test to test against the same. I'd be grateful if you could also test the patch against the repositoryies you mention.
I can confirm that the patch did indeed fix the problem. I was able to convert both repositories successfully (and very quickly, no less), and they both work fine (I did normal development activity with them) with an unmodified Git 2.48 after the migration process.
Thanks again for the quick fix.
Signature · 2 lines
brian m. carlson (they/them or he/him) Toronto, Ontario, CA
Quoted 6 lines · brian m. carlson
Quoted 4 lines · Karthik Nayak
"brian m. carlson" <[email protected]> writes:
I'm attaching a patch below which should fixes the issue for me and also adding a test to test against the same. I'd be grateful if you could also test the patch against the repositoryies you mention.
Quoted 6 lines
I can confirm that the patch did indeed fix the problem. I was able to convert both repositories successfully (and very quickly, no less), and they both work fine (I did normal development activity with them) with an unmodified Git 2.48 after the migration process.
Thanks again for the quick fix.
Thanks, both. Let's merge it down to 'next' and then to 'master', then.
Quoted 13 lines · Junio C Hamano
Quoted 11 lines · brian m. carlson
Quoted 4 lines · Karthik Nayak
"brian m. carlson" <[email protected]> writes:
I'm attaching a patch below which should fixes the issue for me and also adding a test to test against the same. I'd be grateful if you could also test the patch against the repositoryies you mention.I can confirm that the patch did indeed fix the problem. I was able to convert both repositories successfully (and very quickly, no less), and they both work fine (I did normal development activity with them) with an unmodified Git 2.48 after the migration process.
Quoted 4 lines
Quoted 1 line
Thanks again for the quick fix.
Thanks, both. Let's merge it down to 'next' and then to 'master', then.
I was going to send some additional commits around the patch as a v2, but I think they can come in as follow ups. This would ensure that the fix is already in place and we can discuss over the other commits slowly.
Thanks
Quoted 11 lines · brian m. carlson
Quoted 4 lines · Karthik Nayak
"brian m. carlson" <[email protected]> writes:
I'm attaching a patch below which should fixes the issue for me and also adding a test to test against the same. I'd be grateful if you could also test the patch against the repositoryies you mention.I can confirm that the patch did indeed fix the problem. I was able to convert both repositories successfully (and very quickly, no less), and they both work fine (I did normal development activity with them) with an unmodified Git 2.48 after the migration process.
Wohoo!
Quoted 1 line
Thanks again for the quick fix.
Thanks for reporting and testing! I'm glad it fixed the issue.
Quoted 3 lines
--
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA
PATCH v2 1/1 reftable: write correct max_update_index to header
In reply to Karthik NayakIn 297c09eabb (refs: allow multiple reflog entries for the same refname, 2024-12-16), the reftable backend learned to handle multiple reflog entries within the same transaction. This was done modifying the update_index for reflogs with multiple indices. During writing the logs, the max_update_index of the writer was modified to ensure the limits were raised to the modified update_indexs.
However, since ref entries are written before the modification to the max_update_index, if there are multiple blocks to be written, the reftable backend writes the header with the old max_update_index. When all logs are finally written, the footer will be written with the new min_update_index. This causes a mismatch between the header and the footer and causes the reftable file to be corrupted. The existing tests only spawn a single block and since headers are lazily written with the first block, the tests didn't capture this bug.
To fix the issue, the appropriate max_update_index limit must be set even before the first block is written. Add a max_index field to the transaction which holds the max_index within all its updates, then propagate this value to the reftable backend, wherein this is used to the set the max_update_index correctly.
Add a test which creates a few thousand reference updates with multiple reflog entries, which should trigger the bug.
---
Hello,
While this patch was merged to next, Dscho reported that it was flaky on macos pipeline. On further investigation I found this was easily reproducible when the leak sanitizer was turned on and the reftable tests were run. The fix was simply to add the missing 0 initialization.
I also found that we didn't set the appropriate values for all elements of the reftable_transaction_data. Which I've done now.
The patch is based on Maint f93ff170b9 (Git 2.48.1, 2025-01-13).
Thanks
Changes from v1:
1. Correctly intialize the value to 0 for write_transaction_table_arg.max_index.
2. Set the max_index for all elements of reftable_transaction_data.
3. The range diff was made against the commit in Junio's branch gitster/kn/reflog-migration-fix, which included his 'S-o-b', which I've removed in this patch, since that is usually applied by Junio. Not sure if that was the right move.
Range diff:
1: bc67b4ab5f ! 1: 5489826b47 reftable: write correct max_update_index to header
@@ Commit message
Reported-by: brian m. carlson <[email protected]>
Signed-off-by: Karthik Nayak <[email protected]>
- Signed-off-by: Junio C Hamano <[email protected]>
## refs.c ##
@@ refs.c: int ref_transaction_update_reflog(struct ref_transaction *transaction,
@@ refs/reftable-backend.c: struct write_transaction_table_arg {
};
struct reftable_transaction_data {
+@@ refs/reftable-backend.c: static int prepare_transaction_update(struct write_transaction_table_arg **out,
+ arg->updates_nr = 0;
+ arg->updates_alloc = 0;
+ arg->updates_expected = 0;
++ arg->max_index = 0;
+ }
+
+ arg->updates_expected++;
@@ refs/reftable-backend.c: static int write_transaction_table(struct reftable_writer *writer, void *cb_data
struct reftable_log_record *logs = NULL;
struct ident_split committer_ident = {0};
@@ refs/reftable-backend.c: static int write_transaction_table(struct reftable_writ
if (ret < 0)
goto done;
@@ refs/reftable-backend.c: static int reftable_be_transaction_finish(struct ref_store *ref_store UNUSED,
- struct reftable_transaction_data *tx_data = transaction->backend_data;
int ret = 0;
-+ if (tx_data->args)
-+ tx_data->args->max_index = transaction->max_index;
-+
for (size_t i = 0; i < tx_data->args_nr; i++) {
++ tx_data->args[i].max_index = transaction->max_index;
++
ret = reftable_addition_add(tx_data->args[i].addition,
write_transaction_table, &tx_data->args[i]);
+ if (ret < 0)
## t/t1460-refs-migrate.sh ##
@@ t/t1460-refs-migrate.sh: do
---
refs.c | 7 +++++++
refs/refs-internal.h | 1 +
refs/reftable-backend.c | 20 ++++++++++----------
t/t1460-refs-migrate.sh | 12 ++++++++++++
4 files changed, 30 insertions(+), 10 deletions(-)refs.c +7 -0
@@ -1345,6 +1345,13 @@ int ref_transaction_update_reflog(struct ref_transaction *transaction, update->flags &= ~REF_HAVE_OLD ; update->index = index; +/* +* Reference backends may need to know the max index to optimize +* their writes. So we store the max_index on the transaction level. +*/ +if (index > transaction->max_index) + transaction->max_index = index; +return 0 ; }
refs/refs-internal.h +1 -0
@@ -203,6 +203,7 @@ struct ref_transaction {enum ref_transaction_state state;void *backend_data;unsigned int flags; +unsigned int max_index; };/*
refs/reftable-backend.c +10 -10
@@ -942,6 +942,7 @@ struct write_transaction_table_arg {size_t updates_nr;size_t updates_alloc;size_t updates_expected; +unsigned int max_index; };struct reftable_transaction_data {
@@ -1019,6 +1020,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out, arg->updates_nr =0 ; arg->updates_alloc =0 ; arg->updates_expected =0 ; + arg->max_index =0 ; } arg->updates_expected++;
@@ -1428,7 +1430,6 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_datastruct reftable_log_record *logs =NULL ;struct ident_split committer_ident = {0 };size_t logs_nr =0 , logs_alloc =0 , i; -uint64_t max_update_index = ts;const char *committer_info;int ret =0 ;
@@ -1438,7 +1439,12 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_dataQSORT (arg->updates, arg->updates_nr, transaction_update_cmp); -reftable_writer_set_limits (writer, ts, ts); +/* +* During reflog migration, we add indexes for a single reflog with +* multiple entries. Each entry will contain a different update_index, +* so set the limits accordingly. +*/ +reftable_writer_set_limits (writer, ts, ts + arg->max_index);for (i =0 ; i < arg->updates_nr; i++) {struct reftable_transaction_update *tx_update = &arg->updates[i];
@@ -1540,12 +1546,6 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data*/ log->update_index = ts + u->index; -/* -* Note the max update_index so the limit can be set later on. -*/ -if (log->update_index > max_update_index) - max_update_index = log->update_index; - log->refname =xstrdup (u->refname);memcpy (log->value.update.new_hash, u->new_oid.hash,GIT_MAX_RAWSZ );
@@ -1609,8 +1609,6 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data* and log blocks. */ if (logs) { -reftable_writer_set_limits (writer, ts, max_update_index); - ret =reftable_writer_add_logs (writer, logs, logs_nr);if (ret <0 ) goto done;
@@ -1632,6 +1630,8 @@ static int reftable_be_transaction_finish(struct ref_store *ref_store UNUSED,int ret =0 ;for (size_t i =0 ; i < tx_data->args_nr; i++) { + tx_data->args[i].max_index = transaction->max_index; + ret =reftable_addition_add (tx_data->args[i].addition, write_transaction_table, &tx_data->args[i]);if (ret <0 )
t/t1460-refs-migrate.sh +12 -0
@@ -227,6 +227,18 @@ do done done +test_expect_success 'multiple reftable blocks with multiple entries' ' + test_when_finished "rm -rf repo" && + git init --ref-format=files repo && + test_commit -C repo first && + printf "create refs/heads/ref-%d HEAD\n" $(test_seq 5000) >stdin && + git -C repo update-ref --stdin <stdin && + test_commit -C repo second && + printf "update refs/heads/ref-%d HEAD\n" $(test_seq 3000) >stdin && + git -C repo update-ref --stdin <stdin && + test_migration repo reftable +' + test_expect_success 'migrating from files format deletes backend files' ' test_when_finished "rm -rf repo" && git init --ref-format=files repo &&
Signature · 1 line
2.47.0
Re: [PATCH v2] reftable: write correct max_update_index to header
Quoted 4 lines · Karthik Nayak
While this patch was merged to next, Dscho reported that it was flaky on macos pipeline. On further investigation I found this was easily reproducible when the leak sanitizer was turned on and the reftable tests were run. The fix was simply to add the missing 0 initialization.
If it is already _in_ 'next', please turn it into a relative patch on top of it, instead of replacing it.
That will give you an opportunity to describe the breakage in the original version, which everybody missed until it hit 'next'. And you can also credit the folks who reported the breakage, and describe the fix.
The reason we do not revert out of 'next' lightly is because the changes we merge to 'next' are supposed to be reviewed well enough, which means that any bug we discover later is likely to have been caused by mistakes any of us may repeat in the future, and it is worth documenting in our history.
It is quite a different review philosophy if you compare the rules we use for patches that haven't hit 'next'. These uncooked patches may have mistrakes that reviewers can easily spot and get corrected, and these easy ones are not worth documenting as much.
Quoted 1 line
The patch is based on Maint
f93ff170b9(Git 2.48.1, 2025-01-13).
Thanks.
Re: [PATCH v2] reftable: write correct max_update_index to header
Quoted 9 lines · Junio C Hamano
Quoted 4 lines · Karthik Nayak
While this patch was merged to next, Dscho reported that it was flaky on macos pipeline. On further investigation I found this was easily reproducible when the leak sanitizer was turned on and the reftable tests were run. The fix was simply to add the missing 0 initialization.
If it is already _in_ 'next', please turn it into a relative patch on top of it, instead of replacing it.
For now, I have tentatively created the following and will queue on a separate kn/reflog-migration-fix-fix topic (which would be ahead of kn/reflog-migration-fix topic by this one commit), in the hope that it can be replaced with a version with proper commit log message that describes what bugs in the original "fix" are addressed, how they are caused (e.g., how does it lead to the breakage to forget clearing of arg->max_index in the first hunk had?), and what their fixes are.
Thanks.
--- >8 ---
From: Karthik Nayak <[email protected]>
Date: Fri, 20 Dec 2024 13:58:37 +0100
Subject: [PATCH] SQUASH - needs to describe the breakage and fix in v1
--- refs/reftable-backend.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
refs/reftable-backend.c +3 -3
@@ -920,6 +920,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out, arg->updates_nr =0 ; arg->updates_alloc =0 ; arg->updates_expected =0 ; + arg->max_index =0 ; } arg->updates_expected++;
@@ -1502,10 +1503,9 @@ static int reftable_be_transaction_finish(struct ref_store *ref_store UNUSED,struct reftable_transaction_data *tx_data = transaction->backend_data;int ret =0 ; -if (tx_data->args) - tx_data->args->max_index = transaction->max_index; -for (size_t i =0 ; i < tx_data->args_nr; i++) { + tx_data->args[i].max_index = transaction->max_index; + ret =reftable_addition_add (tx_data->args[i].addition, write_transaction_table, &tx_data->args[i]);if (ret <0 )
Signature · 1 line
2.48.1-259-gf9754493bb
PATCH v3 1/1 refs: fix uninitialized memory access of `max_index`
When migrating reflogs between reference backends, maintaining the original order of the reflog entries is crucial. To achieve this, an index field is stored within the ref_update struct.
In the reftable backend, before writing any references, the writer must be configured with the minimum and maximum update index values. The max_update_index is derived from the maximum ref_update.index value in a transaction . The commit bc67b4ab5f (reftable: write correct max_update_index to header, 2025-01-15) addressed this by propagating the max_update_index value from the transaction to write_transaction_table_arg and, ultimately, to reftable_writer_set_limits(), which sets the min and max index for the reftable writer.
However, that commit introduced an issue:
- In reftable_transaction_data, which contains an array of write_transaction_table_arg, only the first element was assigned the max_index value.
As a result, any elements beyond the first in the array contained uninitialized max_index. The writer contains multiple elements of write_transaction_table_arg to correspond to different worktrees being written. This uninitialized value was later used to set the max_update_index for the writer, potentially causing overflow or undefined behavior.
Fix this by:
- Initializing the max_index field to 0.
- Moving the assignment of max_index in reftable_be_transaction_finish() inside the loop, ensuring all elements of the array are correctly initialized.
Initializing max_index to 0 is not strictly necessary, as all elements of write_transaction_table_arg.max_index are now assigned correctly. However, this initialization is added for consistency and to safeguard against potential future changes that might inadvertently introduce uninitialized memory access.
---
Hello,
As suggested, I've redone my patch to make this a relative patch on top of 'kn/reflog-migration-fix'.
This is based on top of maint with 'kn/reflog-migration-fix' merged in.
Thanks
--- refs/reftable-backend.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
refs/reftable-backend.c +3 -3
@@ -1020,6 +1020,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out, arg->updates_nr =0 ; arg->updates_alloc =0 ; arg->updates_expected =0 ; + arg->max_index =0 ; } arg->updates_expected++;
@@ -1628,10 +1629,9 @@ static int reftable_be_transaction_finish(struct ref_store *ref_store UNUSED,struct reftable_transaction_data *tx_data = transaction->backend_data;int ret =0 ; -if (tx_data->args) - tx_data->args->max_index = transaction->max_index; -for (size_t i =0 ; i < tx_data->args_nr; i++) { + tx_data->args[i].max_index = transaction->max_index; + ret =reftable_addition_add (tx_data->args[i].addition, write_transaction_table, &tx_data->args[i]);if (ret <0 )
Signature · 1 line
2.47.0
Re: [PATCH v3] refs: fix uninitialized memory access of `max_index`
Quoted 20 lines · Karthik Nayak
When migrating reflogs between reference backends, maintaining the original order of the reflog entries is crucial. To achieve this, an
indexfield is stored within theref_updatestruct.In the reftable backend, before writing any references, the writer must be configured with the minimum and maximum update index values. The
max_update_indexis derived from the maximumref_update.indexvalue in a transaction . The commitbc67b4ab5f(reftable: write correctmax_update_indexto header, 2025-01-15) addressed this by propagating themax_update_indexvalue from the transaction towrite_transaction_table_argand, ultimately, toreftable_writer_set_limits(), which sets the min and max index for the reftable writer.However, that commit introduced an issue:
- In
reftable_transaction_data, which contains an array ofwrite_transaction_table_arg, only the first element was assigned themax_indexvalue.
Quoted 6 lines
As a result, any elements beyond the first in the array contained uninitialized
max_index. The writer contains multiple elements ofwrite_transaction_table_argto correspond to different worktrees being written. This uninitialized value was later used to set themax_update_indexfor the writer, potentially causing overflow or undefined behavior.
It reads a bit funny as a bulleted list with a single item, only. A suggestion for the above:
However, we only set the update index for the first write_transaction_table_arg, even though there can be multiple such arguments. This is the case when we write to multiple stacks in a single transaction, e.g. when updating references in two different worktrees at once. And, if so, we wouldn't have initialized the update index for any but the first such argument. This uninitialized value was later used to set the max_update_index for the writer, potentially causing undefined behaviour.
Other than that this is nicely described, and the fix looks reasonable, too.
Patrick
Re: [PATCH v3] refs: fix uninitialized memory access of `max_index`
Quoted 40 lines · Patrick Steinhardt
Quoted 20 lines · Karthik Nayak
When migrating reflogs between reference backends, maintaining the original order of the reflog entries is crucial. To achieve this, an
indexfield is stored within theref_updatestruct.In the reftable backend, before writing any references, the writer must be configured with the minimum and maximum update index values. The
max_update_indexis derived from the maximumref_update.indexvalue in a transaction . The commitbc67b4ab5f(reftable: write correctmax_update_indexto header, 2025-01-15) addressed this by propagating themax_update_indexvalue from the transaction towrite_transaction_table_argand, ultimately, toreftable_writer_set_limits(), which sets the min and max index for the reftable writer.However, that commit introduced an issue:
- In
reftable_transaction_data, which contains an array ofwrite_transaction_table_arg, only the first element was assigned themax_indexvalue.Quoted 6 lines
As a result, any elements beyond the first in the array contained uninitialized
max_index. The writer contains multiple elements ofwrite_transaction_table_argto correspond to different worktrees being written. This uninitialized value was later used to set themax_update_indexfor the writer, potentially causing overflow or undefined behavior.It reads a bit funny as a bulleted list with a single item, only. A suggestion for the above:
However, we only set the update index for the first
write_transaction_table_arg, even though there can be multiple such arguments. This is the case when we write to multiple stacks in a single transaction, e.g. when updating references in two different worktrees at once. And, if so, we wouldn't have initialized the update index for any but the first such argument. This uninitialized value was later used to set themax_update_indexfor the writer, potentially causing undefined behaviour.
Quoted 4 lines
Other than that this is nicely described, and the fix looks reasonable, too.
Patrick
Thanks Patrick for taking the time, this seems much better. Let me add this in for the next version.
Re: [PATCH v3] refs: fix uninitialized memory access of `max_index`
Quoted 2 lines · Karthik Nayak
Thanks Patrick for taking the time, this seems much better. Let me add this in for the next version.
I've sent a v4 of this patch, but forgot to set the In-reply-to header. The patch can be found at [1].
Patrick
[1]: <b7e3dd3cc870024f0e80dad26c5a7a96483c6cf4.1737970803.git.ps@pks.im>
Re: [PATCH v3] refs: fix uninitialized memory access of `max_index`
Quoted 10 lines · Patrick Steinhardt
Quoted 2 lines · Karthik Nayak
Thanks Patrick for taking the time, this seems much better. Let me add this in for the next version.
I've sent a v4 of this patch, but forgot to set the In-reply-to header. The patch can be found at [1].
Patrick
[1]: <b7e3dd3cc870024f0e80dad26c5a7a96483c6cf4.1737970803.git.ps@pks.im>
Ah, how very considerate of you. List-archive spelunkers have bidirectional links to see (1) from an older discussion, how the issues were resolved, and (2) from the final attempt, where the issues came from.
Thanks.
Re: [PATCH v2] reftable: write correct max_update_index to header
In reply to Junio C HamanoQuoted 21 lines · Junio C Hamano
Quoted 4 lines · Karthik Nayak
While this patch was merged to next, Dscho reported that it was flaky on macos pipeline. On further investigation I found this was easily reproducible when the leak sanitizer was turned on and the reftable tests were run. The fix was simply to add the missing 0 initialization.
If it is already _in_ 'next', please turn it into a relative patch on top of it, instead of replacing it.
That will give you an opportunity to describe the breakage in the original version, which everybody missed until it hit 'next'. And you can also credit the folks who reported the breakage, and describe the fix.
The reason we do not revert out of 'next' lightly is because the changes we merge to 'next' are supposed to be reviewed well enough, which means that any bug we discover later is likely to have been caused by mistakes any of us may repeat in the future, and it is worth documenting in our history.
Quoted 5 lines
It is quite a different review philosophy if you compare the rules we use for patches that haven't hit 'next'. These uncooked patches may have mistrakes that reviewers can easily spot and get corrected, and these easy ones are not worth documenting as much.
Thanks Junio, I understand your reasoning here and it makes sense to me. Do you think it is worthwhile to also add something like this to our Documentation? I couldn't find anything there. I'll add a small patch to the bottom of this mail.
Quoted 3 lines
Quoted 1 line
The patch is based on Maint
f93ff170b9(Git 2.48.1, 2025-01-13).Thanks.
-- >8 --
Subject: [PATCH] doc: add guideline to tackle bugs in next
When fixing a bug in a topic already merged into next, there are no strict guidelines to follow. While topics in seen can be reverted, topics in next have undergone thorough review. Documenting fixes for such topics is valuable, as it helps to clarify the issue and contributes to preventing similar problems in the future.
--- Documentation/SubmittingPatches | 7 +++++++ 1 file changed, 7 insertions(+)
Documentation/SubmittingPatches +0 -0
@@ -115,6 +115,13 @@ latest HEAD commit of `maint` or `master` based
on the following cases:
new API features on the cutting edge that recently appeared in master but were not available in the released version).
+* If you're fixing a bug in a topic that's already been merged into
+ `next`, it's preferable to create a patch relative to that topic. + This approach allows you to describe the issue in the original version + that went unnoticed until it reached next. Additionally, it provides + an opportunity to credit those who reported the issue and document + the details of the fix.
+
* Otherwise (such as if you are adding new features) use master.
Signature · 1 line
2.47.0
Re: [PATCH v2] reftable: write correct max_update_index to header
Quoted 4 lines · Karthik Nayak
Thanks Junio, I understand your reasoning here and it makes sense to me. Do you think it is worthwhile to also add something like this to our Documentation? I couldn't find anything there. I'll add a small patch to the bottom of this mail.
Perhaps. I think referring people to "Notes from the maintainer" may also work without duplicating information.
Thanks.