Skip to contentSkip to conversation

Support for large files on 32bit systems.

PATCH In review Martin Waitz 4 participants 6 messages Jun 15, 2016 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Support for large files on 32bit systems.

Glibc uses the same size for int and off_t by default.
In order to support large pack sizes (>2GB) we force Glibc to a 64bit off_t.

---
 git-compat-util.h |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)
git-compat-util.h +5 -0
index bf3ceb8..38f9594 100644
@@ -19,6 +19,11 @@
 #define _GNU_SOURCE 1
 #define _BSD_SOURCE 1
 
+/* support large files even on 32bit systems */
+#ifdef __GLIBC__
+#define _FILE_OFFSET_BITS=64
+#endif
+
 #include <unistd.h>
 #include <stdio.h>
 #include <sys/stat.h>
Signature · 4 lines
1.5.0.rc2.g08361-dirty

-- 
Martin Waitz
#1 lore raw

hoi :)

arg, this patch is completely broken.
I tested it by setting the macro via the command line and the = is still left in there. And __GLIBC__ is not yet defined at this point.

However, we need to set _FILE_OFFSET_BITS for glibc somehow. What is the best way to do so?

Signature · 1 line
Martin Waitz
#2 lore raw
Quoted 7 lines · Martin Waitz

arg, this patch is completely broken.
I tested it by setting the macro via the command line and the = is still left in there. And __GLIBC__ is not yet defined at this point.

However, we need to set _FILE_OFFSET_BITS for glibc somehow. What is the best way to do so?

I think the only way to do with this is to have the Makefile detect if -D_FILE_OFFSET_BITS=64 is required to be added to CFLAGS based on some rule (e.g. uname output?), then add that to CFLAGS when needed.

This is one area where autoconf is nice, as you can compile a thousand little C programs to see how the compiler reacts by default, and when you attempt to set this define, and then decide if you should include it or not.

It is also the part of autoconf that's not-so-nice... as it compiles a thousand little C programs just to figure out how to compile the real program. ;-)

Signature · 1 line
Shawn.
#3 lore raw

hoi :)

Quoted 3 lines · Shawn O. Pearce

I think the only way to do with this is to have the Makefile detect if -D_FILE_OFFSET_BITS=64 is required to be added to CFLAGS based on some rule (e.g. uname output?), then add that to CFLAGS when needed.

something like this?

(I'm just testing it with a large data transfer, which will take some time..., will report success later)

+++
Support for large files on 32bit systems.

Glibc uses the same size for int and off_t by default.
In order to support large pack sizes (>2GB) we force Glibc to a 64bit off_t.

---
 Makefile |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
Makefile +2 -0
index ebecbbd..325c19f 100644
@@ -334,9 +334,11 @@ EXTLIBS = -lz
 
 ifeq ($(uname_S),Linux)
 	NO_STRLCPY = YesPlease
+	BASIC_CFLAGS = -D_FILE_OFFSET_BITS=64
 endif
 ifeq ($(uname_S),GNU/kFreeBSD)
 	NO_STRLCPY = YesPlease
+	BASIC_CFLAGS = -D_FILE_OFFSET_BITS=64
 endif
 ifeq ($(uname_S),Darwin)
 	NEEDS_SSL_WITH_CRYPTO = YesPlease
Signature · 5 lines
1.5.0.80.g42d14


-- 
Martin Waitz
#4 lore raw
In reply to Shawn O. Pearce
Quoted 6 lines · Shawn O. Pearce
Quoted 5 lines · Martin Waitz

arg, this patch is completely broken.
I tested it by setting the macro via the command line and the = is still left in there. And __GLIBC__ is not yet defined at this point.

Quoted 6 lines
Quoted 2 lines

However, we need to set _FILE_OFFSET_BITS for glibc somehow. What is the best way to do so?

I think the only way to do with this is to have the Makefile detect if -D_FILE_OFFSET_BITS=64 is required to be added to CFLAGS based on some rule (e.g. uname output?), then add that to CFLAGS when needed.

Why not simply defining _FILE_OFFSET_BITS=64 unconditionally?

It certainly won't cause GIT to explode if compiled against something else than glibc.

Nicolas

#5 lore raw
Quoted 8 lines · Nicolas Pitre
Quoted 3 lines

I think the only way to do with this is to have the Makefile detect if -D_FILE_OFFSET_BITS=64 is required to be added to CFLAGS based on some rule (e.g. uname output?), then add that to CFLAGS when needed.

Why not simply defining _FILE_OFFSET_BITS=64 unconditionally?

It certainly won't cause GIT to explode if compiled against something else than glibc.

I like the simplicity of that approach. Like this?

git-compat-util.h +2 -0
index 105ac28..dd5f369 100644
@@ -1,6 +1,8 @@
 #ifndef GIT_COMPAT_UTIL_H
 #define GIT_COMPAT_UTIL_H
 
+#define _FILE_OFFSET_BITS 64
+
 #ifndef FLEX_ARRAY
 #if defined(__GNUC__) && (__GNUC__ < 3)
 #define FLEX_ARRAY 0