Skip to contentSkip to conversation

open files limit

RFC PATCH New Pete Wyckoff 2 participants 4 messages Jun 15, 2016

lore
Share & tools
lore
#0 lore raw

Somebody at $work found this problem:

    $ git ls-files -s | wc
    error: packfile .git/objects/pack/pack-1627e77da82bbb36118762649c8aa88c05664b1e.pack cannot be accessed
    [..lots more similar errors..]

Turns out his shell's open file descriptor limit was 500. And there are 1600 pack files in the repo.

Increasing the descriptor limit to 1024 fixed it. I could probably get him to repack, which may also fix it.

Does it seem feasible to look for EMFILE errors and close some packs? Or at least spit out a more intuitive error?

		-- Pete
#1 lore raw
Quoted 6 lines · Pete Wyckoff

Somebody at $work found this problem:

   $ git ls-files -s | wc
   error: packfile .git/objects/pack/pack-1627e77da82bbb36118762649c8aa88c05664b1e.pack cannot be accessed
   [..lots more similar errors..]
Quoted 8 lines

Turns out his shell's open file descriptor limit was 500.  And there are 1600 pack files in the repo.

Increasing the descriptor limit to 1024 fixed it.  I could probably get him to repack, which may also fix it.

Does it seem feasible to look for EMFILE errors and close some packs?  Or at least spit out a more intuitive error?

What version of Git? I remember fixing this already.... :-)

Signature · 1 line
Shawn.
#2 lore raw

[email protected] wrote on Fri, 12 Aug 2011 09:09 -0700:

Quoted 10 lines · Pete Wyckoff
Quoted 9 lines

Somebody at $work found this problem:

   $ git ls-files -s | wc
   error: packfile .git/objects/pack/pack-1627e77da82bbb36118762649c8aa88c05664b1e.pack cannot be accessed
   [..lots more similar errors..]

Turns out his shell's open file descriptor limit was 500.  And there are 1600 pack files in the repo.

Quoted 7 lines
Quoted 5 lines

Increasing the descriptor limit to 1024 fixed it.  I could probably get him to repack, which may also fix it.

Does it seem feasible to look for EMFILE errors and close some packs?  Or at least spit out a more intuitive error?

What version of Git? I remember fixing this already.... :-)

Initially 1.7.5.4. Same problem on 1.7.6 and master.

I have your "Limit file descriptors used by packs" (c793430, 28 feb 2011).

It fails here:

	if (!is_pack_valid(p)) {
		error("packfile %s cannot be accessed", p->pack_name);
		goto next;
	}

because p->pack_fd is -1, because an earlier git_open_noatime() got EMFILE. The function unuse_one_window() is never able to find anything to close.

I'll do some more debugging this weekend. Thanks for pointing out that it _should_ be fixed.

		-- Pete
#3 lore raw

RFC PATCH 1/1 test showing EMFILE error with too many packs

[email protected] wrote on Fri, 12 Aug 2011 10:26 -0700:

Quoted 31 lines

[email protected] wrote on Fri, 12 Aug 2011 09:09 -0700:

Quoted 10 lines · Pete Wyckoff
Quoted 9 lines

Somebody at $work found this problem:

   $ git ls-files -s | wc
   error: packfile .git/objects/pack/pack-1627e77da82bbb36118762649c8aa88c05664b1e.pack cannot be accessed
   [..lots more similar errors..]

Turns out his shell's open file descriptor limit was 500.  And there are 1600 pack files in the repo.

Quoted 7 lines
Quoted 5 lines

Increasing the descriptor limit to 1024 fixed it.  I could probably get him to repack, which may also fix it.

Does it seem feasible to look for EMFILE errors and close some packs?  Or at least spit out a more intuitive error?

What version of Git? I remember fixing this already.... :-)

Initially 1.7.5.4. Same problem on 1.7.6 and master.

I have your "Limit file descriptors used by packs" (c793430, 28 feb 2011).

It fails here:

	if (!is_pack_valid(p)) {
		error("packfile %s cannot be accessed", p->pack_name);
		goto next;
	}
Quoted 6 lines

because p->pack_fd is -1, because an earlier git_open_noatime() got EMFILE. The function unuse_one_window() is never able to find anything to close.

I'll do some more debugging this weekend. Thanks for pointing out that it _should_ be fixed.

Here's a test case (on master). It is easy to repack the repository to avoid this situation, so I'm not convinced this is a bug.

But it would be nice at least to report what's going on when we run out of file descriptors.

The current error is:

    $ ../git commit -m fileN
    error: packfile .git/objects/pack/pack-7bff59b72a19f371ccd3e4ffefaac6cd1e07c6ed.pack cannot be accessed
    error: invalid object 100644 a7d02de3336c72e6d2d24f382b82195dbf625404 for 'file47'
    error: Error building trees
		-- Pete

--------8<--------

From: Pete Wyckoff <[email protected]>
Date: Sat, 13 Aug 2011 15:57:55 -0400
Subject: [RFC PATCH] test showing EMFILE error with too many packs

In a repository with too many pack files, it is
possible to have too many open at once and exceed
the process file descriptor limit.

This test shows one failure with an artificially low limit of 50. The same failure happens at a "normal" value like 1024, but takes longer.

---
 t/t5308-pack-emfile.sh |   29 +++++++++++++++++++++++++++++
 1 files changed, 29 insertions(+), 0 deletions(-)
 create mode 100755 t/t5308-pack-emfile.sh
t/t5308-pack-emfile.sh new file +29 -0
new file mode 100755
index 0000000..66c62a7
@@ -0,0 +1,29 @@
+#!/bin/sh
+
+test_description='repos with too many packs should not fail'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+	for i in {1..60}
+	do
+		echo "$i" >"file$i" &&
+		git add "file$i" &&
+		test_tick &&
+		git commit -q -m "$i" &&
+		git repack -q
+	done
+	git prune-packed -q
+'
+
+# run in a subshell since limits cannot be increased
+test_expect_failure 'commit fail with too many packs' '
+	(
+	ulimit -n 50 &&
+	>"fileN" &&
+	git add "fileN" &&
+	git commit -q -m "fileN"
+	)
+'
+
+test_done
Signature · 1 line
1.7.5.4