2GB File limitation
Hello,
Is someone working on a way to overcome the 2GB file limitation in Linux? I currently backup several servers using a dedicated hard drive for the backups. Recently I saw one backup die saying the the file size had been exceeded. I've never had good luck with tape backups, yes they backup, but whenever I really need a file, it can't be retrieved.
TIA,
Bobby
Quoted 4 lines · Bobby Hitt
Is someone working on a way to overcome the 2GB file limitation in Linux? I currently backup several servers using a dedicated hard drive for the backups. Recently I saw one backup die saying the the file size had been exceeded.
This is your lucky day. The 2GB limit has been fixed a year or two ago. You need an "LFS" patch for 2.2 kernels (any vendor kernel will have this), or any 2.4 kernel.
You ALSO need to ensure your libc supports large files (glibc-2.2 definitely will), and also that your tools can handle it (needs to be compiled with special LARGEFILE defines, and open files with O_LARGEFILE).
Cheers, Andreas
--
Andreas Dilger \ "If a man ate a pound of pasta and a pound of antipasto,
\ would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/ -- Dogbert
Quoted 6 lines · Bobby Hitt
Is someone working on a way to overcome the 2GB file limitation in Linux? I currently backup several servers using a dedicated hard drive for the backups. Recently I saw one backup die saying the the file size had been exceeded. I've never had good luck with tape backups, yes they backup, but whenever I really need a file, it can't be retrieved.
file sizes up to ~ 2 TB are supported under the 2.4 kernel. (or 2.2 kernel
+ patches) Most utilities are updated to use O_LARGEFILE properly, in any recent 2.4-based Linux distribution. I regularly use 6-10 GB files.
Ingo
Quoted 11 lines · Ingo Molnar
Quoted 6 lines · Bobby Hitt
Is someone working on a way to overcome the 2GB file limitation in Linux? I currently backup several servers using a dedicated hard drive for the backups. Recently I saw one backup die saying the the file size had been exceeded. I've never had good luck with tape backups, yes they backup, but whenever I really need a file, it can't be retrieved.
file sizes up to ~ 2 TB are supported under the 2.4 kernel. (or 2.2 kernel
+ patches) Most utilities are updated to useO_LARGEFILEproperly, in any recent 2.4-based Linux distribution. I regularly use 6-10 GB files.
Like Ingo says, most of modern distributions begin to work with O_LARGEFILE properly, but still some utilities fail it.
Your older backup application program may fail it too.
The solution could be as simple as piping the backup material into another program, which then pipes it out, e.g.:
'|dd bs=1M of=/backups/fileNNN'
Quoted 1 line
Ingo
/Matti Aarnio
Do you know if they ever fixed this same issue in Microsoft Access? :-) Does Linux support tape drives? :-)
Sure, Linux needs to be able to handle files > 2 GB, but I wonder if backups over a network to a hard drive are giving you a false sense of "security". In the "enterprise" world, you have to be *religious* about backups. Incrementals at least daily, fulls at least weekly, all backups verified and *off-site secure storage of backup media*. Dang! The soap company wants their &$^%&$# box back :-).
--
M. Edward (Ed) Borasky, Chief Scientist, Borasky Research
http://www.borasky-research.net http://www.aracnet.com/~znmeb
mailto:[email protected] mailto:[email protected]
If there's nothing to astrology, how come so many famous men were born on holidays?
Quoted 29 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]]On Behalf Of Bobby Hitt
Sent: Thursday, September 27, 2001 11:17 PM
To: [email protected]
Subject: 2GB File limitationHello,
Is someone working on a way to overcome the 2GB file limitation in Linux? I
currently backup several servers using a dedicated hard drive for the backups. Recently I saw one backup die saying the the file size had been exceeded. I've never had good luck with tape backups, yes they backup, but whenever I really need a file, it can't be retrieved.TIA,
Bobby
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 7 lines · Bobby Hitt
Hello,
Is someone working on a way to overcome the 2GB file limitation in Linux? I currently backup several servers using a dedicated hard drive for the backups. Recently I saw one backup die saying the the file size had been exceeded. I've never had good luck with tape backups, yes they backup, but whenever I really need a file, it can't be retrieved.
The problem has been fixed in the file system, the kernel and
the supporting libraries for ages. Applications may need to be updated to take advantage of 64 bit file offsets and file operations, though. You didn't say how you were backing up those other servers. Is it a particular backup program? If so, that program may need to be rebuilt or updated. Are you backing up over NFS or some other file transport where the Linux box is just acting as a file server or is the backup program running on that Linux box?
Oh... And what version of Linux? If it's old enough, I
suppose you might still have a problem there. Recent (RedHat 7.x and contemporaries) distributions shouldn't be a problem.
Quoted 3 lines
TIA,
Bobby
Mike
Signature · 4 lines
Michael H. Warfield | (770) 985-6132 | [email protected] (The Mad Wizard) | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0xDF1DD471 | possible worlds. A pessimist is sure of it!
Quoted 5 lines
Is someone working on a way to overcome the 2GB file limitation in Linux? I currently backup several servers using a dedicated hard drive for the backups. Recently I saw one backup die saying the the file size had been exceeded. I've never had good luck with tape backups, yes they backup, but whenever I really need a file, it can't be retrieved.
I use the -B flag to insure the archives are just under 2GB. This generates multiple volumes by appending a sequence number if necessary.
For example, on a /home dump of >2GB, the commands
ddate=date +%m%d%y-%H%M%S
dump -0uM -b 32 -B 2097142 -f /mnt/home.$ddate.L0.dump -L "abit:/home" /home
generate
/mnt/home.092801-130439.L0.dump001/mnt/home.092801-130439.L0.dump002
each guaranteed to be 10K under 2GB. Fixes potential portability issues also.
rgds,
tim.
--