Code reorgnization

Git's top directory is crowded and I think it's agreed that moving test-* to t/helper is a good move. I just wanted to check if we could take this opportunity (after v2.8.0) to move some other files too. I propose the following new subdirs

lib
---
This contains files that are about data structures or algorithms. Very general purpose. This directory includes

argv-array.[ch] base85.c column.[ch] delta.h diff-delta.c hashmap.[ch] hex.c khash.h kwset.[ch] levenshtein.[ch] mergesort.[ch] patch-delta.c prio-queue.[ch] sha1-array.[ch] sha1-lookup.[ch] strbuf.[ch] string-list.[ch] url.[ch] urlmatch.[ch] utf8.[ch] varint.[ch] versioncmp.c wildmatch.[ch]

odb
---
The grouping of object database files is to easily make connections between them. Unlike, for example, diff-related files which either start with "diff" or has that word in the file name to make connections.

alloc.c blob.[ch] bulk-checkin.[ch] commit-slab.h commit.[ch] object.[ch] pack.h pack-revindex.[ch] replace_object.c sha1_file.c streaming.[ch] tag.[ch] tree.[ch]

index
-----
For the same reason of odb subdir. This directory contains

cache-tree.[ch] name-hash.c preload-index.c read-cache.c split-index.[ch] unpack-trees.[ch]

sys (or maybe util or support)
------------------------------
These are still general purpose but is usually system-related. They are still far away from git's core logic. I want to separate them to make it easier to spot "important" files at top dir.

abspath.c color.[ch] copy.c csum-file.[ch] ctype.c date.c editor.c exec_cmd.[ch] gettext.[ch] gettext.h gpg-interface.[ch] ident.c lockfile.[ch] mailinfo.[ch] mailmap.[ch] pager.c parse-options-cb.c parse-options.[ch] pathspec.[ch] pkt-line.[ch] progress.[ch] prompt.[ch] quote.[ch] run-command.[ch] sideband.[ch] sigchain.[ch] symlinks.c tar.h tempfile.[ch] thread-utils.[ch] trace.[ch] unix-socket.[ch] usage.c userdiff.[ch] wrapper.c write_or_die.c zlib.c

Good? Bad? Ugly?
--
Duy

I wonder whether previous discussions on this still count? See:

http://marc.info/?l=git&m=129650572621523&w=1

-- Thomas Adam

If you refer to ancient discussion, especially to a large thread like that one, please spend a bit more time to summarize it. It is between one person spends a bit more time, and all others independently go there and read.

The essense of the proposal [1] back then was to move all the source file to src/, rename t/ to testsuite. And I think [2] is a pretty good summary of the common feeling back then that explains why the proposal died out:

Moving everything into src/ and calling it "organized" doesn't actually accomplish much other than perhaps making the README file more visible to newbs; things are _still_ a mess, just a mess with four more letters...

This round is slightly more organized, so many points the old thread raised would not apply, I suspect.

[References]

*1* http://thread.gmane.org/gmane.comp.version-control.git/165720/focus=165748

*2* http://thread.gmane.org/gmane.comp.version-control.git/165720/focus=166019

For now I would just go with 3 directories:

non-git/ (or util, helpers, or anything that could be ripped out and be useful
e.g. strbufs, argv-array run-command, lockfile
git/ (maybe called lib? All stuff that is pure Git and is used for libgit

builtin/ (as we have it today + all that stuff that doesn't go into git/ very well?)

Thanks,
Stefan

It is unclear where you want to have standalone programs in the above. I'd say lib/ and src/ for the first two, where lib/ is for things that could be lifted without any Git dependencies and src/ for everything else.

Aren't there some folks who link directly with our codebase (I am thinking about cgit, but hjemli.net/git/cgit does not seem to be responding anymore)?

CGit lives at https://git.zx2c4.com/cgit/ these days.

The organisation of the git code shouldn't make a difference since CGit just links with libgit.a, even if it does CGit pulls in git.git as a submodule so it can just fix any problems in the same commit that updates the submodule reference.

I was mostly worried about where Duy and Stefan want to place *.h

*.h stay with their *.c. CFLAGS has two more -Isrc and -Ilib. I don't expect any #include line changes. Maybe we can start moving stuff to "lib" soon. Many of them rarely receive changes these days. The creation of "src" could be more disruptive and can wait until $(topdir) is once again unbearable.

Duy

They now have a new mailing list.[1] I guess they forward all the mails to the new one[2]. The new one seems active. [3]

[1] : [email protected]
[2] : https://lists.zx2c4.com/pipermail/cgit/2013-May/001380.html
[3] : http://news.gmane.org/gmane.comp.version-control.cgit

Hi Duy,

The name "lib" makes it sound as if this contains the source code of libgit.a. Maybe "generic" or "common" or "util" would be better (my favorite would be "util").

Disruptive. Probably a change for 3.0?

Ciao,
Dscho

We tested it with the builtin rename a long time ago, so it's probably not bad. By the principle of "dogfooding", we should try it soon and make sure it's not disruptive, or prepare ourselves for such a change (I think git-am can't track renames, for example, without us giving it a clue somehow)

Duy

Too fine-grained to induce confusion for things that have to work as a bridge between two categories (e.g. odb & index). In short, bad and ugly.

I am OK with a looser classification e.g. (1) things that can be used without Git at all like strbuf, string-list vs (2) things that are Git but shared across different subcommands like read-cache, sha1_file, vs (3) command implementations (e.g. builtin/ and also standalone).

I guess I don't really see the "crowded" problem, but perhaps that is because I am more or less familiar with where things are in git's code base. I suppose if you were looking for a "utility" function, you might look in "util" and therefore have a smaller set of files to check.

But I think we also run into the opposite problem: I am looking for some particular function, but I can't find it, because I am looking in "util" and it is in some other directory. And when files move around, it makes history harder to follow (maybe that is because git sucks and we need to make it better, but certainly I run into mild annoyances with the builtin/ rename when digging in history).

And you have a similar problem when creating new files. Which slot do they go in? What if they could feasibly go into two slots?

So there can be friction either way. In practice I find I just use ctags to jump to the functions I am interested in, and I don't care that much about filenames.

The reorganization that _would_ be more interesting to me is not files in directories, but rather functions in files. I wish everything were designed more as modules with a pair of matching ".c" and ".h" files, with a public interface defined in the ".h", and messier, private stuff in the ".c". But we have some real dumping grounds:

1. cache.h has the declarations for at least a dozen different modules; besides being hard to navigate, it causes more frequent recompilation than necessary.

2. a few of the .c files could probably be split (e.g., dir.c is where all of the pathspec code lives, even though that is used for much more than filesystem access these days).

Splitting those up would _also_ introduce friction (and actually worse than whole-file renames, because finding code movement between files is an even harder / more expensive problem). But I feel like it would buy a lot more in terms of code clarity, and in reducing the scope of code which has access to private, static interfaces.

-Peff

Yeah, for finding a particular function, I just "git grep" (or rgrep from emacs) if I fail to locate it after the first guess. We have this problem nowadays anyway. Besides builtin, we also have ewah, refs and some more subdirs.

Everything goes to topdir (or later on "src") by default and only goes to "lib" when it's _obvious_ that it's disconnected from git (i'm talking about the "lib/src" layout).

Heh.. that's what I wanted to do (or at least discuss) after files are moved :)

.. and this is why I did not raise it in the first mail.

Duy