File Limit in Kernel?

I have a directory with 39,000 files in it, and I'm trying to use the cp command to copy them into another directory, and neither the cp or the mv command will work, they both same "argument list too long" when I use:

cp -f * /usr/local/www/images

or

mv -f * /usr/local/www/images

Is this a kernel limitation? If yes, how can I get around it? If no, anyone know a workaround? I appreciate it.

Adam Voigt ([email protected])
The Cryptocomm Group
My GPG Key: http://64.238.252.49:8080/adam_at_cryptocomm.asc

Note that this is not a kernel related question. The * in the command line is expanded into a list of all entries in the current directory, which results in a command line longer than allowed. Try this instead:

find -maxdepth 1 -print0 | \
	xargs -0 --replace=% cp -f % /usr/local/www/images

--Andreas.

Find has an -exec operator in the first place, so this is a little:

find -maxdepth 1 -exec cp -f '{}' /usr/local/www/images ';'

-------------------------------------------------------------------------------
Jan 'Bulb' Hudec <[email protected]>

Jan Hudec <[email protected]> writes:

|> On Tue, Nov 12, 2002 at 04:57:20PM +0100, Andreas Gruenbacher wrote:
|> > On Tuesday 12 November 2002 16:38, Adam Voigt wrote:
|> > > I have a directory with 39,000 files in it, and I'm trying to use the cp
|> > > command to copy them into another directory, and neither the cp or the
|> > > mv command will work, they both same "argument list too long" when I
|> > > use:
|> > >
|> > > cp -f * /usr/local/www/images
|> > >
|> > > or
|> > >
|> > > mv -f * /usr/local/www/images
|> > 
|> > Note that this is not a kernel related question.

Actually it is, because it's a kernel limit. Userspace does not have this problem in general.

|> > expanded into a list of all entries in the current directory, which results 
|> > in a command line longer than allowed. Try this instead:
|> > 
|> > find -maxdepth 1 -print0 | \
|> > 	xargs -0 --replace=% cp -f % /usr/local/www/images
|> 
|> Find has an -exec operator in the first place, so this is a little:
|> 
|> find -maxdepth 1 -exec cp -f '{}' /usr/local/www/images ';'

Or even using the shell:

for f in *; do cp -f "$f" /usr/local/www/image; done

Andreas.

Andreas Schwab, SuSE Labs, [email protected]
SuSE Linux AG, Deutschherrnstr. 15-19, D-90429 Nürnberg
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

No, it is not a kernel limit, it is a limit to your shell (bash, for example). Look at xargs to get around it.

A related limit is that the popular ext2 and 3 file systems get inefficient when directories have so many files. The work-around for that is to have your files either hashed or organized across a collection of directories.

-kb

E2BIG The argument list is too big.

_SC_ARG_MAX
The maximum length of the arguments to the exec() family of functions; the corresponding macro is ARG_MAX.

On my RedHat 8.0 box:

       $ getconf ARG_MAX
       131072

It is definately a kernel limitation, although as other people have pointed out, there are common userspace solutions to the problem.

mark

[email protected]/[email protected]/[email protected] __________________________
.  .  _  ._  . .   .__    .  . ._. .__ .   . . .__  | Neighbourhood Coder
|\/| |_| |_| |/    |_     |\/|  |  |_  |   |/  |_   | 
|  | | | | \ | \   |__ .  |  | .|. |__ |__ | \ |__  | Ottawa, Ontario, Canada

  One ring to rule them all, one ring to find them, one ring to bring them all
                       and in the darkness bind them...

                           http://mark.mielke.cc/

The '*' is expanded by your shell to be a command-line that has 39,000 file-names in it! It is probably way too long for a command- line (argument list).

The easiest way is to do:

mv -f a* /usr/local/www/images
mv -f b* /usr/local/www/images
mv -f c* /usr/local/www/images

... until the remaining argument list is short enough for the '*' only.

You can also do a loop in a shell if the shell's internal buffer is big enough for the expanded '*' ...

for x in * ; do mv -f $x /usr/local/www/images ; done

Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Bush : The Fourth Reich of America

It's not a kernel limitiation, its a shell limitation. "*" expands to the list of 39000 names, which is too large.

You could try something like:

ls .|xargs cp -f --target-directory=/usr/local/www/images

Chris

Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]

Yes, but you can get around it in userspace. See
http://www.linuxjournal.com/article.php?sid=6060

(Short answer is this from include/linux/binfmts.h.:
/*
* MAX_ARG_PAGES defines the number of pages allocated for arguments
* and envelope for the new program. 32 should suffice, this gives
* a maximum env+arg of 128kB w/4KB pages!
*/
#define MAX_ARG_PAGES 32

I'm assuming your arch has 4kb pages, so the 'problem' is that you're passing more than 128kb of cmdline arguments.)

The article has a bunch of ways. You don't really need to change the kernel though ... if you don't mind generating 39000 new processes one after the other, I'd do something like 'for FILE in * ; do mv $FILE /usr/local/www/images ; done'. Probably slower than calling mv once due to process overhead done 39000 times, but it *works*, and it's simple. (If you use a csh instead of something bash-like, change that to fit.)

Matt

the * is expanded to a string, which is too long to be handed as a single command line argument. Think it is in the kernel, yes. You can solve this by using find|xargs or tar|tar or by specifying a directory.

Greetings
Bernd

Sort of, there is a limit to how many k of arguments you can have on a command line. Having grown up with much smaller limits in early UNIX, I got into the habit of using xargs when I wasn't sure. You can avoid one exec per file behaviour by something like:
ls | xargs -l50 echo | xargs -i mv "{}" destdir

You can also do useful things by using find and the -p option of cpio.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.