git checkout crashes after server being updated to Debian X86_64

PATCH In review Raffael Reichelt 4 participants 7 messages Oct 18, 2016 – Oct 19, 2016

Share & tools

Raffael Reichelt

author

2016-10-18 15:17 UTC #0 lore raw

Hello!

I have a serious problem with git, After my provider had updated to a X86_64 architecture git crashes with various memory-related errors. This is happening remote when pushing to the repository from my local machine as well as trying it on a shell on the server itself.

This are the error-messages:

fatal: Out of memory, realloc failed
fatal: recursion detected in die handler
fatal: recursion detected in die handler

or
fatal: unable to create threaded lstat
fatal: recursion detected in die handler
or
fatal: unable to create threaded lstat
*** Error in `git': double free or corruption (fasttop): 0x0000000000a8ade0 ***
fatal: recursion detected in die handler
Aborted

It’s obviously not a problem of the repository - happens with all of them. I think it is also not a question of size - happens with a 80M Repository as well as with a 500M one.

Any way: did a

git fsck
Prüfe Objekt-Verzeichnisse: 100% (256/256), Fertig. Prüfe Objekte: 100% (56305/56305), Fertig.

git gc --auto --prune=today —aggressive
git repack

Additionally I played around some config parameters so my config now looks like:
[http]

    postbuffer = 524288000

[pack]
threads = 1
deltaCacheSize = 128m
packSizeLimit = 128m
windowMemory = 128m
[core]
packedGitLimit = 128m
packedGitWindowSize = 128m
repositoryformatversion = 0
filemode = true
bare = true

I am running
git version 2.1.4

on
Linux infongp-de65 3.14.0-ui16196-uiabi1-infong-amd64 #1 SMP Debian 3.14.73-2~ui80+4 (2016-07-13) x86_64 GNU/Linux

Anyone out there to help me getting out of this trouble?

Regards,
Raffael

René Scharfe

2016-10-18 16:42 UTC #1 lore raw

Am 18.10.2016 um 17:17 schrieb Raffael Reichelt:

Quoted 50 lines

Hello!
I have a serious problem with git, After my provider had updated to a X86_64 architecture git crashes with various memory-related errors. This is happening remote when pushing to the repository from my local machine as well as trying it on a shell on the server itself.
This are the error-messages:
fatal: Out of memory, realloc failed
fatal: recursion detected in die handler
fatal: recursion detected in die handler
or
fatal: unable to create threaded lstat
fatal: recursion detected in die handler
or
fatal: unable to create threaded lstat
*** Error in `git': double free or corruption (fasttop): 0x0000000000a8ade0 ***
fatal: recursion detected in die handler
Aborted
It’s obviously not a problem of the repository - happens with all of them. I think it is also not a question of size - happens with a 80M Repository as well as with a 500M one.
Any way: did a
git fsck
Prüfe Objekt-Verzeichnisse: 100% (256/256), Fertig. Prüfe Objekte: 100% (56305/56305), Fertig.
git gc --auto --prune=today —aggressive
git repack
Additionally I played around some config parameters so my config now looks like:
[http]
    postbuffer = 524288000
[pack]
threads = 1
deltaCacheSize = 128m
packSizeLimit = 128m
windowMemory = 128m
[core]
packedGitLimit = 128m
packedGitWindowSize = 128m
repositoryformatversion = 0
filemode = true
bare = true

Quoted 7 lines

I am running
git version 2.1.4
on
Linux infongp-de65 3.14.0-ui16196-uiabi1-infong-amd64 #1 SMP Debian 3.14.73-2~ui80+4 (2016-07-13) x86_64 GNU/Linux
Anyone out there to help me getting out of this trouble?

Git 2.1.4 is the version that comes with Debian stable according to https://packages.debian.org/jessie/git, so I guess using a more recent version is not a reasonable option.

What do "file $(which git)" and "ulimit -a" return? Do you have an x86-64 binary and no unnecessarily low limits set?

René

Raffael Reichelt

author

2016-10-18 17:00 UTC #2 lore raw

Hello Renè!
file is returning

/usr/bin/git: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=ee62e538d6fe6673d3ba49f0e66bfec784cc32bc, stripped

and ulimit is:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 1
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 512
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) 1800
max user processes (-u) 42
virtual memory (kbytes, -v) 786432
file locks (-x) unlimited

Support told me git is limited to 600M

Regrads,
Rafael

Quoted 66 lines

Am 18.10.2016 um 18:42 schrieb René Scharfe <[email protected]>:
Am 18.10.2016 um 17:17 schrieb Raffael Reichelt:
Quoted 50 lines
Hello!
I have a serious problem with git, After my provider had updated to a X86_64 architecture git crashes with various memory-related errors. This is happening remote when pushing to the repository from my local machine as well as trying it on a shell on the server itself.
This are the error-messages:
fatal: Out of memory, realloc failed
fatal: recursion detected in die handler
fatal: recursion detected in die handler
or
fatal: unable to create threaded lstat
fatal: recursion detected in die handler
or
fatal: unable to create threaded lstat
*** Error in `git': double free or corruption (fasttop): 0x0000000000a8ade0 ***
fatal: recursion detected in die handler
Aborted
It’s obviously not a problem of the repository - happens with all of them. I think it is also not a question of size - happens with a 80M Repository as well as with a 500M one.
Any way: did a
git fsck
Prüfe Objekt-Verzeichnisse: 100% (256/256), Fertig. Prüfe Objekte: 100% (56305/56305), Fertig.
git gc --auto --prune=today —aggressive
git repack
Additionally I played around some config parameters so my config now looks like:
[http]
postbuffer = 524288000
[pack]
threads = 1
deltaCacheSize = 128m
packSizeLimit = 128m
windowMemory = 128m
[core]
packedGitLimit = 128m
packedGitWindowSize = 128m
repositoryformatversion = 0
filemode = true
bare = true
Quoted 7 lines
I am running
git version 2.1.4
on
Linux infongp-de65 3.14.0-ui16196-uiabi1-infong-amd64 #1 SMP Debian 3.14.73-2~ui80+4 (2016-07-13) x86_64 GNU/Linux
Anyone out there to help me getting out of this trouble?
Git 2.1.4 is the version that comes with Debian stable according to https://packages.debian.org/jessie/git, so I guess using a more recent version is not a reasonable option.
What do "file $(which git)" and "ulimit -a" return? Do you have an x86-64 binary and no unnecessarily low limits set?
René

Duy Nguyen

2016-10-19 14:28 UTC #3 lore raw

In reply to Raffael Reichelt

Quoted 9 lines · Raffael Reichelt

Hello!
I have a serious problem with git, After my provider had updated to a X86_64 architecture git crashes with various memory-related errors. This is happening remote when pushing to the repository from my local machine as well as trying it on a shell on the server itself.
This are the error-messages:
fatal: Out of memory, realloc failed
fatal: recursion detected in die handler
fatal: recursion detected in die handler

You other mail said memory is capped at 600MB, which should be a lot for normal repositories. If you set the environment variable GIT_ALLOC_LIMIT to maybe 500MB or lower (convert it to kilobytes first) and git attempts to allocate more than that (just that one time, not total mem) then it's caught and we get a glimpse of how much memory git may need. Unfortunately we can't get a stack trace or anything like that unless you rebuild Git from source.

Quoted 3 lines

or
fatal: unable to create threaded lstat
fatal: recursion detected in die handler

Hmm.. with "max user processes (-u) 42" we should be fine because we only create 20 threads max. What happens if you set core.preloadindex to false? Can it run until the end or hit some other fatal errors?

There's room for improvement in preload_index(). If we hit resource limit like this, it's not the end of the world and we should be able to keep going. But threaded lstat has been available for a long time and this is the first time I see a report like this, not sure if it's worth fixing.

Signature · 1 line

Duy

Raffael Reichelt

author

2016-10-19 14:13 UTC #4 lore raw

Quoted 15 lines

Am 19.10.2016 um 15:27 schrieb Duy Nguyen <[email protected]>:
Quoted 9 lines · Raffael Reichelt
Hello!
I have a serious problem with git, After my provider had updated to a X86_64 architecture git crashes with various memory-related errors. This is happening remote when pushing to the repository from my local machine as well as trying it on a shell on the server itself.
This are the error-messages:
fatal: Out of memory, realloc failed
fatal: recursion detected in die handler
fatal: recursion detected in die handler

Quoted 7 lines

You other mail said memory is capped at 600MB, which should be a lot for normal repositories. If you set the environment variable GIT_ALLOC_LIMIT to maybe 500MB or lower (convert it to kilobytes first) and git attempts to allocate more than that (just that one time, not total mem) then it's caught and we get a glimpse of how much memory git may need. Unfortunately we can't get a stack trace or anything like that unless you rebuild Git from source.

This was no change: crashed with the same errors …

Quoted 9 lines

Quoted 4 lines
or
fatal: unable to create threaded lstat
fatal: recursion detected in die handler
Hmm.. with "max user processes (-u) 42" we should be fine because we only create 20 threads max. What happens if you set core.preloadindex to false? Can it run until the end or hit some other fatal errors?

This did the trick :) I just repeatedly did a forced checkout and it went until the end without errors THX a lot!

Raffael

Duy Nguyen

2016-10-19 16:21 UTC #5 lore raw

PATCH 1/1 Add a knob to abort on die() (was Re: git checkout crashes after ...)

In reply to Duy Nguyen

Quoted 5 lines · Duy Nguyen

If you set the environment variable GIT_ALLOC_LIMIT ... git attempts to allocate more than that ... then it's caught and we get a glimpse of how much memory git may need. Unfortunately we can't get a stack trace or anything like that unless you rebuild Git from source.

It's moments like this that I wish we had a knob to force core dumps. And I often modify die_builtin() to add '*(char*)0 = 1;' to force a core dump when I can't figure out some problem based on the error message alone.

So.. how about we do something like this? We could extend it to abort on error() as well as die(). Aborting on warning() may be a bit too much though. On glibc systems we could even print the back trace without aborting, which helps in some cases.

The long variable name and value are on purpose to hopefully not trigger this by mistake.

git.c +19 -0

index ab5c99c..5fea224 100644
@@ -622,15 +622,34 @@ static int run_argv(int *argcp, const char ***argv)
 	return done_alias;
 }
 
+static NORETURN void die_by_aborting(const char *err, va_list params)
+{
+	vreportf("fatal: ", err, params);
+	abort();
+}
+
+static NORETURN void die_silently_by_aborting(const char *err, va_list params)
+{
+	abort();
+}
+
 int cmd_main(int argc, const char **argv)
 {
 	const char *cmd;
 	int done_help = 0;
+	const char *die_abort_env = getenv("GIT_ABORT_ON_FATAL_ERRORS");
 
 	cmd = argv[0];
 	if (!cmd)
 		cmd = "git-help";
 
+	if (die_abort_env) {
+		if (!strcmp(die_abort_env, "yes please"))
+			set_die_routine(die_by_aborting);
+		else if (!strcmp(die_abort_env, "just die"))
+			set_die_routine(die_silently_by_aborting);
+	}
+
 	trace_command_performance(argv);
 
 	/*

--
Duy

Jeff King

2016-10-19 21:15 UTC #6 lore raw

Re: [PATCH] Add a knob to abort on die() (was Re: git checkout crashes after ...)

Quoted 12 lines · Duy Nguyen

Quoted 5 lines · Duy Nguyen
If you set the environment variable GIT_ALLOC_LIMIT ... git attempts to allocate more than that ... then it's caught and we get a glimpse of how much memory git may need. Unfortunately we can't get a stack trace or anything like that unless you rebuild Git from source.
It's moments like this that I wish we had a knob to force core dumps. And I often modify die_builtin() to add '*(char*)0 = 1;' to force a core dump when I can't figure out some problem based on the error message alone.

Quoted 4 lines

So.. how about we do something like this? We could extend it to abort on error() as well as die(). Aborting on warning() may be a bit too much though. On glibc systems we could even print the back trace without aborting, which helps in some cases.

I have been tempted by something like this, too, and have occasionally resorted to patching in an abort() call to get cores from sub-processes.

I'm not sure how useful it would be in practice in deployed versions of git, though. You'd have to coach the user into finding the core file and generating the backtrace from it. Something that called backtrace(3) automatically would be more seamless (but provides worse information than gdb), and otherwise it is probably just as easy to ship the user cut-and-paste instructions to use gdb.

See the previous discussion in this subthread:

http://public-inbox.org/git/[email protected]/T/#u

-Peff