Skip to contentSkip to conversation

Big file support

DISCUSSION Rick Stevens 11 participants 11 messages Feb 27, 2002 – Feb 28, 2002

lore
Share & tools
lore
#0 lore raw

I'm not certain if this is the right place, but are there plans to have big file support (files >2GB) anytime soon? I ask, as we use Linux to serve LOTS of streaming media and the logs for popular sites often exceed 2GB. I'd like to see the ability to handle at least 16GB files, possibly more.

Please cc: me on any replies if possible. I've been REALLY busy and am finding it hard to keep up with l-k traffic.

Thanks!
----------------------------------------------------------------------
- Rick Stevens, SSE, VitalStream, Inc. [email protected] -
- 949-743-2010 (Voice) http://www.vitalstream.com -
- -
- Never eat anything larger than your head -
----------------------------------------------------------------------

#1 lore raw

This support has been in ReiserFS for some time. On a SuSE system running 2.4.16 (their official kernel) you can issue: mkreiserfs -v2 /dev/sdaxxx and there you go, LFS is in ver2.

Regards,

jon
----- Original Message -----
From: "Rick Stevens" <[email protected]>
To: "Linux-Kernel" <[email protected]>
Sent: Wednesday, February 27, 2002 11:37 AM
Subject: Big file support

Quoted 23 lines

I'm not certain if this is the right place, but are there plans to have big file support (files >2GB) anytime soon? I ask, as we use Linux to serve LOTS of streaming media and the logs for popular sites often exceed 2GB. I'd like to see the ability to handle at least 16GB files, possibly more.

Please cc: me on any replies if possible. I've been REALLY busy and am finding it hard to keep up with l-k traffic.

Thanks!
----------------------------------------------------------------------
- Rick Stevens, SSE, VitalStream, Inc. [email protected] -
- 949-743-2010 (Voice) http://www.vitalstream.com -

(quoted diff) +0 -4
-                                                                    -
-              Never eat anything larger than your head              -
----------------------------------------------------------------------
 
-

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
In reply to Rick Stevens

A lot of the kernel supports big files already. The real problem is the fact that the primary Linux file system, ext3, does not. If you use some file system besides ext3, big files should work.

Linux already has API calls to read big files.

-- Barubary

----- Original Message -----
From: "Rick Stevens" <[email protected]>
To: "Linux-Kernel" <[email protected]>
Sent: Wednesday, February 27, 2002 11:37 AM
Subject: Big file support

Quoted 22 lines

I'm not certain if this is the right place, but are there plans to have big file support (files >2GB) anytime soon? I ask, as we use Linux to serve LOTS of streaming media and the logs for popular sites often exceed 2GB. I'd like to see the ability to handle at least 16GB files, possibly more.

Please cc: me on any replies if possible. I've been REALLY busy and am finding it hard to keep up with l-k traffic.

Thanks!
----------------------------------------------------------------------
- Rick Stevens, SSE, VitalStream, Inc. [email protected] -
- 949-743-2010 (Voice) http://www.vitalstream.com -

(quoted diff) +0 -4
-                                                                    -
-              Never eat anything larger than your head              -
----------------------------------------------------------------------
 
-

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#3 lore raw
Quoted 3 lines

A lot of the kernel supports big files already. The real problem is the fact that the primary Linux file system, ext3, does not. If you use some file system besides ext3, big files should work.

This is incorrect information. Ext3 supports large files. Whoever told you otherwise was wrong.

#5 lore raw
In reply to Rick Stevens
Quoted 3 lines · Barubary

A lot of the kernel supports big files already. The real problem is the fact that the primary Linux file system, ext3, does not. If you use some file system besides ext3, big files should work.

Bullshit. EXT2/EXT3 does support large files. Has done so since
kernel 1.2 in fact, altough formely only at 64 bit machines.

Since 2.4 the kernel has been changed internally so that it supports large files also at measly 32 bit thingies including i386...

There are several filesystems which are 64-bit/large-file supporting, but also some which are inherently incapable to exceed 2G or 4G.

It looks like the LOOP driver lands in between -- it should be LFS capable, but it isn't.

Quoted 1 line

Linux already has API calls to read big files.

Those were done for 2.4 kernel too.

Quoted 37 lines

-- Barubary

----- Original Message -----
From: "Rick Stevens" <[email protected]>
To: "Linux-Kernel" <[email protected]>
Sent: Wednesday, February 27, 2002 11:37 AM
Subject: Big file support

Quoted 18 lines

I'm not certain if this is the right place, but are there plans to have big file support (files >2GB) anytime soon? I ask, as we use Linux to serve LOTS of streaming media and the logs for popular sites often exceed 2GB. I'd like to see the ability to handle at least 16GB files, possibly more.

Please cc: me on any replies if possible. I've been REALLY busy and am finding it hard to keep up with l-k traffic.

Thanks!
----------------------------------------------------------------------
- Rick Stevens, SSE, VitalStream, Inc. [email protected] -
- 949-743-2010 (Voice) http://www.vitalstream.com -

(quoted diff) +0 -4
-                                                                    -
-              Never eat anything larger than your head              -
----------------------------------------------------------------------
 
-
Quoted 4 lines

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#6 lore raw

Re: Big file support\

In reply to Rick Stevens

On Wed, Feb 27, 2002 at 10:34:26PM +0200, Matti Aarnio wrote:

It looks like the LOOP driver lands in between -- it should be LFS capable, but it isn't.

Really?

I used to use loop with large lop files all the time, I had to fix the utils that mounted/attached the loop device though, as they would fail without proper LFS build smarts.

--cw

#7 lore raw
In reply to Rick Stevens
Quoted 2 lines · Rick Stevens

I'm not certain if this is the right place, but are there plans to have big file support (files >2GB) anytime soon?

This has been supported for well over a year.

Rik

Signature · 4 lines
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/
#8 lore raw
In reply to Rick Stevens
Quoted 5 lines · Rick Stevens

I'm not certain if this is the right place, but are there plans to have big file support (files >2GB) anytime soon? I ask, as we use Linux to serve LOTS of streaming media and the logs for popular sites often exceed 2GB. I'd like to see the ability to handle at least 16GB files, possibly more.

Check http://www.suse.de/~aj/linux_lfs.html,

Andreas

Signature · 4 lines
 Andreas Jaeger
  SuSE Labs [email protected]
   private [email protected]
    http://www.suse.de/~aj
#4 lore raw

Re: Big file support (emperical evidence)

In reply to Alan Cox

I agree -- and I was actually surprised. On Tuesday I wrote a script that created some huge files on an ext3 filesystem, expecting it to die at 2GB, but it didn't die until I passed 8GB (and filled the partition). (( redhat 7.2 ))

Quoted 6 lines · Alan Cox
Quoted 3 lines

A lot of the kernel supports big files already. The real problem is the fact that the primary Linux file system, ext3, does not. If you use some file system besides ext3, big files should work.

This is incorrect information. Ext3 supports large files. Whoever told you otherwise was wrong.

Signature · 4 lines
Stephen Samuel +1(604)876-0426                [email protected]
		   http://www.bcgreen.com/~samuel/
Powerful committed communication, reaching through fear, uncertainty and
doubt to touch the jewel within each person and bring it to life.
#9 lore raw
In reply to Rick Stevens
Quoted 5 lines · Matti Aarnio

There are several filesystems which are 64-bit/large-file supporting, but also some which are inherently incapable to exceed 2G or 4G.

It looks like the LOOP driver lands in between -- it should be LFS capable, but it isn't.

Loop is LFS capable. I know that we shipped LFS enabled loop utilities for 7.2, probably 7.1 as well. They were missed in the first batch of LFS conversions, and several distributions are lagging behind in this area.

-ben

#10 lore raw
In reply to Rick Stevens
Quoted 8 lines · Rick Stevens

I'm not certain if this is the right place, but are there plans to have big file support (files >2GB) anytime soon? I ask, as we use Linux to serve LOTS of streaming media and the logs for popular sites often exceed 2GB. I'd like to see the ability to handle at least 16GB files, possibly more.

Please cc: me on any replies if possible. I've been REALLY busy and am finding it hard to keep up with l-k traffic.

You must be really behind, large file support has been in the current kernel for ~14 months. Of course if your application isn't compiled with LFS enabled it doesn't matter, or if it keeps offsets in long instead of offset types...

Signature · 3 lines
bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.