Skip to contentSkip to conversation

invalid outdated temporary files

RFC PATCH In review Han Xin 2 participants 3 messages Mar 23, 2022 Documentation

lore
Share & tools
lore

Han Xin

author
#0 lore raw

cover invalid outdated temporary files

From: Han Xin <[email protected]>

When I use git on network storage, there may be some temporary files left in git repositories due to the program quitting unexpectedly, and they also prevent me from continuing to use the git command line until I manually delete them.

For some obviously expired temporary files, manual deletion is an extra expensive job, and one that I think can be avoided. So I wrote this patch. When creating a temporary file with the same name, if the mtime of the target file exceeds one day, unlink it first, and then try to recreate it.

Han Xin (1):
tempfile: invalid outdated temporary files

 Documentation/config/core.txt |  6 ++++++
 cache.h                       |  1 +
 config.c                      |  5 +++++
 environment.c                 |  1 +
 t/t3210-pack-refs.sh          | 10 ++++++++++
 tempfile.c                    | 21 +++++++++++++++++++++
 6 files changed, 44 insertions(+)
Signature · 1 line
2.35.1.55.gde4365fc52.agit.6.5.7
author
#1 lore raw

RFC PATCH 1/1 tempfile: invalid outdated temporary files

From: Han Xin <[email protected]>

When git quits unexpectedly, some temporary files(e.g. HEAD.lock, packed-refs.lock, packed-refs.new) may remain in the repository. These files will prevent us from performing the corresponding operations again, even if they were created a long time ago before, until we manually remove them.

In order for git to automatically fix this situation, let's add a config named "core.tempfileExpire". When an attempt is made to create a temporary file that exists and is older than this config value, the file will be unlinked and recreated.

---
 Documentation/config/core.txt |  6 ++++++
 cache.h                       |  1 +
 config.c                      |  5 +++++
 environment.c                 |  1 +
 t/t3210-pack-refs.sh          | 10 ++++++++++
 tempfile.c                    | 21 +++++++++++++++++++++
 6 files changed, 44 insertions(+)
Documentation/config/core.txt +6 -0
index c04f62a54a..9907c9e6f3 100644
@@ -424,6 +424,12 @@ be delta compressed, but larger binary media files won't be.
 +
 Common unit suffixes of 'k', 'm', or 'g' are supported.
 
+core.tempfileExpire::
+	When an attempt is made to create a temporary file that exists
+	and is older than this config value, the file will be unlinked
+	and recreated. This feature is used to ignore temporary files
+	(e.g. *.lock and *.new) remaining due to abnormal exits.
+
 core.excludesFile::
 	Specifies the pathname to the file that contains patterns to
 	describe paths that are not meant to be tracked, in addition
cache.h +1 -0
index 0bc0a37cec..4264fe7960 100644
@@ -972,6 +972,7 @@ extern size_t packed_git_window_size;
 extern size_t packed_git_limit;
 extern size_t delta_base_cache_limit;
 extern unsigned long big_file_threshold;
+extern const char *tempfile_expire;
 extern unsigned long pack_size_limit_cfg;
 
 /*
config.c +5 -0
index e78397725c..36a5bc1960 100644
@@ -1518,6 +1518,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
 		return 0;
 	}
 
+	if (!strcmp(var, "core.tempfileexpire")) {
+		git_config_get_expiry(var, &tempfile_expire);
+		return 0;
+	}
+
 	if (!strcmp(var, "core.packedgitlimit")) {
 		packed_git_limit = git_config_ulong(var, value);
 		return 0;
environment.c +1 -0
index fb55bf6129..20b928e71b 100644
@@ -48,6 +48,7 @@ size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
 size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
 size_t delta_base_cache_limit = 96 * 1024 * 1024;
 unsigned long big_file_threshold = 512 * 1024 * 1024;
+const char *tempfile_expire = "1.day.ago";
 int pager_use_color = 1;
 const char *editor_program;
 const char *askpass_program;
t/t3210-pack-refs.sh +10 -0
index 577f32dc71..7d443c1cfd 100755
@@ -231,6 +231,16 @@ test_expect_success 'timeout if packed-refs.lock exists' '
 	test_must_fail git pack-refs --all --prune
 '
 
+test_expect_success 'success if packed-refs.lock expires' '
+	LOCK=.git/packed-refs.lock &&
+	>"$LOCK" &&
+	test_when_finished "rm -f $LOCK" &&
+	test-tool chmtime -86000 $LOCK &&
+	test_must_fail git pack-refs --all --prune &&
+	test-tool chmtime -400 $LOCK &&
+	git pack-refs --all --prune
+'
+
 test_expect_success 'retry acquiring packed-refs.lock' '
 	LOCK=.git/packed-refs.lock &&
 	>"$LOCK" &&
tempfile.c +21 -0
index 94aa18f3f7..f18e0121d0 100644
@@ -51,6 +51,7 @@
  */
 
 #include "cache.h"
+#include "date.h"
 #include "tempfile.h"
 #include "sigchain.h"
 
@@ -137,6 +138,26 @@ struct tempfile *create_tempfile_mode(const char *path, int mode)
 	strbuf_add_absolute_path(&tempfile->filename, path);
 	tempfile->fd = open(tempfile->filename.buf,
 			    O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC, mode);
+	if (tempfile->fd < 0 && errno == EEXIST) {
+		struct stat st;
+		int err = errno;
+		if (lstat(tempfile->filename.buf, &st) < 0) {
+			warning_errno(_("failed to stat %s"),
+				      tempfile->filename.buf);
+			errno = err;
+		} else if (st.st_mtime <= approxidate(tempfile_expire)) {
+			/*
+			 * If the file is older than core.tmpfileExpire, it is
+			 * probably outdated.  We'll try to remove it and try
+			 * again.
+			 */
+			warning(_("remove outdated tempfile %s"), tempfile->filename.buf);
+			unlink_or_warn(tempfile->filename.buf);
+			tempfile->fd = open(tempfile->filename.buf,
+					    O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC,
+					    mode);
+		}
+	}
 	if (O_CLOEXEC && tempfile->fd < 0 && errno == EINVAL)
 		/* Try again w/o O_CLOEXEC: the kernel might not support it */
 		tempfile->fd = open(tempfile->filename.buf,
Signature · 1 line
2.35.1.55.gde4365fc52.agit.6.5.7
#2 lore raw

Re: [RFC PATCH 1/1] tempfile: invalid outdated temporary files

Quoted 9 lines · Han Xin

From: Han Xin <[email protected]>

When git quits unexpectedly, some temporary files(e.g. HEAD.lock, packed-refs.lock, packed-refs.new) may remain in the repository. These files will prevent us from performing the corresponding operations again, even if they were created a long time ago before, until we manually remove them.

In order for git to automatically fix this situation, let's add a config

I am not sure if we want to automatically "fix" this situation in the first place.

Unless the user is sure of the reason why these lock files are left behind (e.g. "ah, I saw a power failure last night while I was trying to commit"), we are looking at a potential data loss.