Skip to contentSkip to conversation

[Fwd: Re: Implementing a file counter (like "ls | wc")]

Fwd: Re: Implementing a file counter (like "ls | wc") Luciano Moreira - igLnx 5 participants 7 messages Apr 7, 2004 – Apr 8, 2004

lore
Share & tools
lore
#0 lore raw

Does exist another way to detect a directory without stat() ?

Luciano

Quoted 47 lines · Holger Kiehl
Quoted 15 lines · Luciano Moreira - igLnx

-------------- THE CODE HAVE THIS STRUCTURE:

while ((pFile=readdir(pDir))!=NULL) {
    sprintf(szBuf, "%s/%s", pPath, pFile->d_name);
    stat(szBuf, &statFile);
    if (S_ISDIR(statFile.st_mode))  /// LOOK THAT: We don't use recursive
searching, we count only files at current directory excluding others
directories.
       continue;

    /* Filtering */
    if (nNeedFilter) {
        //// I DONT HAVE THE CODE OF FILTERING NOW
        //// BUT I CAN SEND IT LATER IF NECESSARY
       }
Quoted 5 lines

}
-------------- CODE FINISH HERE

Don't use sprintf(), its very expansive. Before the while loop put a pointer after path and / and then strcpy(ptr, pFile->d_name)

stat() is _very_ epansive! It means physical IO and fills up a structure with things you just don't need. If you really do need to filter out directories from your result do the stat after it has passed the filter.

Quoted 6 lines

Does have another mechanism to filter without using of strcmp() / memcmp() ? How if we dont know the size of extension (.c, .cpp, .teste, .longextension, and so on). ?

Compare them yourself with a pointer byte for byte. But the speed gain will not be so high as when you leave away the stat() call.

Holger

#1 lore raw

You should read the GNU source code for ls, they actually filter out directories in a quite interesting maner.

instead of using stat which is quite expensive, they try to open each file as a directory, which is expensive, but not as expensive as stat. If a file opens as a directory, then it is, and they treat it as one, (in your case closing it and ignoring it); else if it fails to open as a directory, they then use stat to get information about it to print;

a simple counter could look something like (my code is paraphrasing parts of the GNU code, but is not a copy) Also treat it as Sudo code that just looks a lot like C as it is untested (not even compiled) and is only trying to make the point.

___________________________________________________
int count = 0;
struct dirent dir_entry;
DIR* directory, test;
char buff[512];

directory = opendir('/etc');
/* check that it opened*/
...

while( dir_entry = readdir(directory) ) {
        strncpy(buff, '/etc/', 6);
        strcat(buff, dir_entry->d_name);
        if( test = opendir( buff ) ) {
                /* gets here the its a directory */ 
                closedir(test);
        } else {
                /* gets here not a directory */ 
                /* code to test if its a regular file */
                count++;
        }
}

closedir(directory);

Quoted 58 lines · Luciano Moreira - igLnx

Does exist another way to detect a directory without stat() ?

Luciano

Quoted 39 lines · Holger Kiehl
Quoted 15 lines · Luciano Moreira - igLnx

-------------- THE CODE HAVE THIS STRUCTURE:

while ((pFile=readdir(pDir))!=NULL) {
    sprintf(szBuf, "%s/%s", pPath, pFile->d_name);
    stat(szBuf, &statFile);
    if (S_ISDIR(statFile.st_mode))  /// LOOK THAT: We don't use recursive
searching, we count only files at current directory excluding others
directories.
       continue;

    /* Filtering */
    if (nNeedFilter) {
        //// I DONT HAVE THE CODE OF FILTERING NOW
        //// BUT I CAN SEND IT LATER IF NECESSARY
       }
Quoted 5 lines

}
-------------- CODE FINISH HERE

Don't use sprintf(), its very expansive. Before the while loop put a pointer after path and / and then strcpy(ptr, pFile->d_name)

stat() is _very_ epansive! It means physical IO and fills up a structure with things you just don't need. If you really do need to filter out directories from your result do the stat after it has passed the filter.

Quoted 6 lines

Does have another mechanism to filter without using of strcmp() / memcmp() ? How if we dont know the size of extension (.c, .cpp, .teste, .longextension, and so on). ?

Quoted 8 lines

Compare them yourself with a pointer byte for byte. But the speed gain will not be so high as when you leave away the stat() call.

Holger

-
To unsubscribe from this list: send the line "unsubscribe linux-c-programming" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 9 lines · John T. Williams

You should read the GNU source code for ls, they actually filter out directories in a quite interesting maner.

instead of using stat which is quite expensive, they try to open each file as a directory, which is expensive, but not as expensive as stat. If a file opens as a directory, then it is, and they treat it as one, (in your case closing it and ignoring it); else if it fails to open as a directory, they then use stat to get information about it to print;

Indeed this is a good idea, but even then I would only go through this code path after the filtering. Any system call is always more expensive then the filtering.

Quoted 5 lines

a simple counter could look something like (my code is paraphrasing parts of the GNU code, but is not a copy) Also treat it as Sudo code that just looks a lot like C as it is untested (not even compiled) and is only trying to make the point.

Another minor improvement would be as stated earlier to put the strncpy() outside the while loop:

Quoted 5 lines

___________________________________________________
int count = 0;
struct dirent dir_entry;
DIR* directory, test;
char buff[512];

char *ptr

Quoted 6 lines

directory = opendir('/etc');
/* check that it opened*/
...

strcpy(buff, '/etc/');
ptr = buff + 5;

Quoted 1 line

while( dir_entry = readdir(directory) ) {

         if (filter fits)
         {
         strcpy(ptr, dir_entry->d_name);
Quoted 8 lines
        if( test = opendir( buff ) ) {
                /* gets here the its a directory */ 
                closedir(test);
        } else {
                /* gets here not a directory */ 
                /* code to test if its a regular file */
                count++;
        }
         }
Quoted 4 lines

}

closedir(directory);

If you have a directory with many files this might make a minor difference. Also note that we now have a strcpy() and not a strcat() in the loop. A strcat() will always have to find the end of the string first before it can do its job.

But most important always try to avoid system calls!

Holger

#3 lore raw

Hi,

I've read all of the mailing thread about this subject and i'd like to ask
some questions..

Quoted 1 line

But most important always try to avoid system calls!

I guess eveybody is accepting that the syscalls is slowing down the process
when invoking them from the critical parts of the code (i hope i understand properly).. I'm trying to write a basic shell for me (just for fun, nothing serious), i'm setting up my user's shell in passwd file to my basic shell's compiled binary, and when i'm logging in from the console, normally i'm falling to a prompt which is written by me.

I'm not using the well-known programs such as 'ls', 'cd' etc.. i have my own
envoriment variables and my own basic programs those are invoking from my prompt with fork and execvp. If i do not want to use the syscalls such as readdir, opendir what i must do?

With readdir, i can read everything from the every type of file system which
supported by kernel *transparently*, for example when i'm working on a disk partition which is formatted with ext3 file system i don't have to try to read the superblocks and inode's adresses from disk because VFS takes care all of the low level operations via readdir (and kernel's modules for that file system) for me, isn't it? If i want to do all of them by my self (sure, this is not necessary but i just want to try and learn) which is the right way i have to go? I thought that it would be a very simple and absolute way looking into the glibc for implementations of the functions like readdir, but nearly i'm sure that it would be impossible to undrstand (and this is another problem, i can't understand the source code usually :( it would be great to hear your advices about how i improve my vision).

Thanks for your time from now,
Regards.

Signature · 12 lines
- -- -- -- -- -- -- -- -- -- -- -- -- -- -
 A. Murat Eren
 [email protected], [email protected]
 http://zion.comu.edu.tr/~evreniz/
- -- -- -- -- -- -- -- -- -- -- -- -- -- -

--
 free software is a matter of liberty,
 not price. to understand the concept,
 you should think of "free speech",
 not "free beer".
-
#4 lore raw

I just want to say after much testing, I've gone and proved myself wrong.

Anyone interested in playing with the concept I've included the code.

Quoted 35 lines · A. Murat Eren

Hi,

I've read all of the mailing thread about this subject and i'd like to ask
some questions..

Quoted 1 line

But most important always try to avoid system calls!

I guess eveybody is accepting that the syscalls is slowing down the process
when invoking them from the critical parts of the code (i hope i understand properly).. I'm trying to write a basic shell for me (just for fun, nothing serious), i'm setting up my user's shell in passwd file to my basic shell's compiled binary, and when i'm logging in from the console, normally i'm falling to a prompt which is written by me.

I'm not using the well-known programs such as 'ls', 'cd' etc.. i have my own
envoriment variables and my own basic programs those are invoking from my prompt with fork and execvp. If i do not want to use the syscalls such as readdir, opendir what i must do?

With readdir, i can read everything from the every type of file system which
supported by kernel *transparently*, for example when i'm working on a disk partition which is formatted with ext3 file system i don't have to try to read the superblocks and inode's adresses from disk because VFS takes care all of the low level operations via readdir (and kernel's modules for that file system) for me, isn't it? If i want to do all of them by my self (sure, this is not necessary but i just want to try and learn) which is the right way i have to go? I thought that it would be a very simple and absolute way looking into the glibc for implementations of the functions like readdir, but nearly i'm sure that it would be impossible to undrstand (and this is another problem, i can't understand the source code usually :( it would be great to hear your advices about how i improve my vision).

Thanks for your time from now,
Regards.

#5 lore raw
In reply to A. Murat Eren
Quoted 5 lines · A. Murat Eren

I've read all of the mailing thread about this subject and i'd like to ask
some questions..

Quoted 1 line

But most important always try to avoid system calls!

Quoted 8 lines

I guess eveybody is accepting that the syscalls is slowing down the process
when invoking them from the critical parts of the code (i hope i understand properly).. I'm trying to write a basic shell for me (just for fun, nothing serious), i'm setting up my user's shell in passwd file to my basic shell's compiled binary, and when i'm logging in from the console, normally i'm falling to a prompt which is written by me.

I'm not using the well-known programs such as 'ls', 'cd' etc.

"cd" isn't a program; it's a built-in shell command. It has to be; a process can't change the current directory of another process, so a "cd" program just wouldn't work.

Quoted 4 lines

. i have my own
envoriment variables and my own basic programs those are invoking from my prompt with fork and execvp. If i do not want to use the syscalls such as readdir, opendir what i must do?

In most of the situations where you use syscalls, you can't avoid using them; the point is not to use them excessively.

E.g. if you're reading a file (or any other descriptor), calling read() a few times and reading a large amount of data each time is more efficient than calling read() lots of times and reading a small amount of data each time.

Or, for an "ls" replacement, only call stat() (or lstat()) if you actually need the additional data (e.g. for "ls -l", "ls -F" etc) rather than always calling it "just in case".

Quoted 4 lines

With readdir, i can read everything from the every type of file system which
supported by kernel *transparently*, for example when i'm working on a disk partition which is formatted with ext3 file system i don't have to try to read the superblocks and inode's adresses from disk because VFS takes care

Quoted 8 lines

all of the low level operations via readdir (and kernel's modules for that file system) for me, isn't it? If i want to do all of them by my self (sure, this is not necessary but i just want to try and learn) which is the right way i have to go? I thought that it would be a very simple and absolute way looking into the glibc for implementations of the functions like readdir, but nearly i'm sure that it would be impossible to undrstand (and this is another problem, i can't understand the source code usually :( it would be great to hear your advices about how i improve my vision).

There is seldom any point in bypassing libc. For a syscall, the libc function will usually consist of the syscall itself, plus setting errno if the syscall returns an error status, so there wouldn't be anything to gain from making the syscall directly.

BTW, on Linux, glibc's readdir() function isn't implemented using the readdir syscall but using the getdents syscall instead. The latter reads multiple directory entries per call, so is more efficient.

You can't realistically bypass the syscalls themselves; not only would trying to read files by accessing the device directly require sufficient privilege, it would also be error-prone, as a user-space process can't hold kernel locks and can't prevent itself being preempted at an inconvenient point.

Signature · 1 line
Glynn Clements <[email protected]>
#6 lore raw

Hi,

Quoted 3 lines

"cd" isn't a program; it's a built-in shell command. It has to be; a process can't change the current directory of another process, so a "cd" program just wouldn't work.

Sorry about that, yes cd is a built-in function of shell. btw, i'm not using
it to change any program's working dir. i'm filling up my shell's current working dir environment variable with calling it to able to use commands like clone of ls without any extra parameter. So for example when i invoke my listing program to see what is in my current working directory, it already knows that i'm in /usr/src from the CWD envoriment var.

Quoted 11 lines

In most of the situations where you use syscalls, you can't avoid using them; the point is not to use them excessively.

E.g. if you're reading a file (or any other descriptor), calling read() a few times and reading a large amount of data each time is more efficient than calling read() lots of times and reading a small amount of data each time.

Or, for an "ls" replacement, only call stat() (or lstat()) if you actually need the additional data (e.g. for "ls -l", "ls -F" etc) rather than always calling it "just in case".

Thank you, i've got the point completely.

Quoted 4 lines

There is seldom any point in bypassing libc. For a syscall, the libc function will usually consist of the syscall itself, plus setting errno if the syscall returns an error status, so there wouldn't be anything to gain from making the syscall directly.

Thank you very much for your advices and time :)

Regards.

Signature · 14 lines
- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -
 A. Murat Eren
 [email protected], [email protected]
 http://zion.comu.edu.tr/~evreniz/
 0x88FD9FC7, 
 910A FCB3 2AAB 4CA5 E4D9 EFFA 6555 A33A 88FD 9FC7
- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -

--
 free software is a matter of liberty,
 not price. to understand the concept,
 you should think of "free speech",
 not "free beer".
-