Skip to contentSkip to conversation

What is an efficient way to get all blobs / trees that have notes attached?

DISCUSSION Sebastian Schuberth 2 participants 5 messages Jun 15, 2016 – Jun 16, 2016

lore
Share & tools
lore
#0 lore raw

Hi,

I'm curious whether there's a more efficient way to get a list of blobs / trees (and their names) that have notes attached than doing this:

1) Get all notes refs I'm interested in (git-for-each-ref).

2) For each notes ref, get the list of notes (git-notes list) and store them in a hash table that maps object hashes to notes.

3) Recursively list all blobs / trees (git-ls-tree) and look whether an object's hash is conatined in our table to get its notes.

In particular 3) could be expensive for repos with a lot of files as we're looking at all of them just to see whether they have notes attached.

Regards,
Sebastian

#1 lore raw
Quoted 7 lines · Sebastian Schuberth

Hi,

I'm curious whether there's a more efficient way to get a list of blobs / trees (and their names) that have notes attached than doing this:

1) Get all notes refs I'm interested in (git-for-each-ref).

Quoted 8 lines

2) For each notes ref, get the list of notes (git-notes list) and store them in a hash table that maps object hashes to notes.

3) Recursively list all blobs / trees (git-ls-tree) and look whether an object's hash is conatined in our table to get its notes.

In particular 3) could be expensive for repos with a lot of files as we're looking at all of them just to see whether they have notes attached.

In (3), why would you need to search through _all_ blobs/trees? Would it not be cheaper to simply query the object type of each annotated object from (2)? I.e. something like:

for notes_ref in $(git for-each-ref refs/notes | cut -c 49-) do

    echo "--- $notes_ref ---"
    for annotated_obj in $(git notes --ref=$notes_ref list | cut -c 41-)
    do
        type=$(git cat-file -t "$annotated_obj")
        if test "$type" != "commit"
        then
            echo "$annotated_obj: $type"
        fi
    done

done

Can probably be made even faster by using the --batch option to cat-file...

...Johan

Signature · 2 lines
Johan Herland, <[email protected]>
www.herland.net
#2 lore raw
Quoted 11 lines · Johan Herland

for notes_ref in $(git for-each-ref refs/notes | cut -c 49-) do

    echo "--- $notes_ref ---"
    for annotated_obj in $(git notes --ref=$notes_ref list | cut -c 41-)
    do
        type=$(git cat-file -t "$annotated_obj")
        if test "$type" != "commit"
        then
            echo "$annotated_obj: $type"
        fi
    done
Quoted 3 lines

done

Can probably be made even faster by using the --batch option to cat-file...

For example:

for notes_ref in $(git for-each-ref refs/notes | cut -c 49-) do

    echo "--- $notes_ref ---"
    git notes --ref=$notes_ref list | cut -c 42- | git cat-file

--batch-check="%(objecttype) %(objectname)" | grep '^\(\(blob\)\|\(tree\)\) '
done

...Johan

Signature · 2 lines
Johan Herland, <[email protected]>
www.herland.net
#3 lore raw
In reply to Johan Herland
Quoted 21 lines · Johan Herland
Quoted 5 lines

3) Recursively list all blobs / trees (git-ls-tree) and look whether an object's hash is conatined in our table to get its notes.

In particular 3) could be expensive for repos with a lot of files as we're looking at all of them just to see whether they have notes attached.

In (3), why would you need to search through _all_ blobs/trees? Would it not be cheaper to simply query the object type of each annotated object from (2)? I.e. something like:

for notes_ref in $(git for-each-ref refs/notes | cut -c 49-) do

    echo "--- $notes_ref ---"
    for annotated_obj in $(git notes --ref=$notes_ref list | cut -c 41-)
    do
        type=$(git cat-file -t "$annotated_obj")
        if test "$type" != "commit"
        then
            echo "$annotated_obj: $type"
        fi
    done
Quoted 1 line

done

Thanks for the idea. The problem is that I do want to list the notes by path of the object they belong to. As a blob could potentially belong to more than one path (copies of files in the repo), I do not see another way of getting that information other than iterating over all blobs and checking what path(s) they belong to.

Signature · 1 line
Sebastian Schuberth
#4 lore raw
Quoted 24 lines · Sebastian Schuberth
Quoted 21 lines · Johan Herland
Quoted 5 lines

3) Recursively list all blobs / trees (git-ls-tree) and look whether an object's hash is conatined in our table to get its notes.

In particular 3) could be expensive for repos with a lot of files as we're looking at all of them just to see whether they have notes attached.

In (3), why would you need to search through _all_ blobs/trees? Would it not be cheaper to simply query the object type of each annotated object from (2)? I.e. something like:

for notes_ref in $(git for-each-ref refs/notes | cut -c 49-) do

    echo "--- $notes_ref ---"
    for annotated_obj in $(git notes --ref=$notes_ref list | cut -c 41-)
    do
        type=$(git cat-file -t "$annotated_obj")
        if test "$type" != "commit"
        then
            echo "$annotated_obj: $type"
        fi
    done
Quoted 1 line

done

Quoted 5 lines

Thanks for the idea. The problem is that I do want to list the notes by path of the object they belong to. As a blob could potentially belong to more than one path (copies of files in the repo), I do not see another way of getting that information other than iterating over all blobs and checking what path(s) they belong to.

True; fundamentally what you want is a blob/tree ID -> path(s) mapping, which is an independent problem, unrelated to to the initial notes lookup.

I don't know of a solution faster than the brute-force search you already sketched. If this lookup is important to your use case, you could consider building/caching the required mapping when the notes are added in the first place, but I don't know if that is possible in your scenario...

...Johan

Quoted 2 lines

--
Sebastian Schuberth

Signature · 2 lines
Johan Herland, <[email protected]>
www.herland.net