readdir & bonnie++
Hi all,
I was testing our fs project using the bonnie++ tool and the following issue came up.
Bonnie++ creates a directory with 16384 files. Then it calls readdir in a loop to read all the entries and calls stat for each. The test fails since only 16383 entries were returned (one is missing). According to my logs our readdir calls filldir on the last item as well. Also strace says that all entries were returned. But the last one is missing. Moreover if I use ls -l on the same directory, all 16384 entries are returned. I don't know what is the difference between the readdir loop in bonnie++ and in ls, but I expect a similar loop. Perhaps using getdents direcly?
Thanks for any suggestions,
cheers,
Tomas
Quoted 5 lines · Tomas Hruby
Bonnie++ creates a directory with 16384 files. Then it calls readdir in a loop to read all the entries and calls stat for each. The test fails since only 16383 entries were returned (one is missing). According to my logs our readdir calls filldir on the last item as well. Also strace says that all entries were returned. But the last one is missing.
Possibly unrelated, but I saw something similar in my filesystem. A simple loop calling readdir would fail, but explicit calls to getdents worked properly (and whatever ls did also worked). In my case, I was playing games with d_off (filp->fpos) that caused d_off to overflow 32 bits. Then glibc would detect this and be "smart" -- seek back to some point and start over to read up to the overflow. Eliminating the overflows fixed it.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines
Quoted 5 lines
Bonnie++ creates a directory with 16384 files. Then it calls readdir in a loop to read all the entries and calls stat for each. The test fails since only 16383 entries were returned (one is missing). According to my logs our readdir calls filldir on the last item as well. Also strace says that all entries were returned. But the last one is missing.
Quoted 6 lines
Possibly unrelated, but I saw something similar in my filesystem. A simple loop calling readdir would fail, but explicit calls to getdents worked properly (and whatever ls did also worked). In my case, I was playing games with
d_off(filp->fpos) that causedd_offto overflow 32 bits. Then glibc would detect this and be "smart" -- seek back to some point and start over to read up to the overflow. Eliminating the
Thanks for a good hint, I checked that but there is no overflow.
Quoted 6 lines
Are you getting this problem *after* bonnie has deleted some entries from the directory? I had some problems with bonnie because it tries to delete entries from a directory in a while(readdir) loop, which happens to test a somewhat obscure and underspecified behaviour of directories. I discussed this on the ML a while back. Maybe this thread might help you:
I check the dicsussion on the suggested thread. It describes all issues I was already facing :-) since we use similar tree structure as ext3.
I return hash (32bit) instead of a real position, but it works fine with ls rm -r so far.
My problem with bonnie happens in a test which counts how many files where created. It just call readdir in loop and counts how many entries were returned. As far as I understand the bonnie code, there were no files deleted yet.
I checked the glibc code (i hope that the correct one) and it seems that the loop inside readdir is returned only if there are no more data in the buffer and so the getdents is called again. My problem is that there SHOULD be the last entry still in the buffer.
Tomas
Quoted 10 lines · Tomas Hruby
My problem with bonnie happens in a test which counts how many files where created. It just call readdir in loop and counts how many entries were returned. As far as I understand the bonnie code, there were no files deleted yet.
I checked the glibc code (i hope that the correct one) and it seems that the loop inside readdir is returned only if there are no more data in the buffer and so the getdents is called again. My problem is that there SHOULD be the last entry still in the buffer.
So, what you're saying is that the getdents() call actually returns the last entry, but readdir() somehow ignores that last entry? If that is the case, I would look at the d_off value for that last entry for weirdness.
Quoted 1 line
Tomas
-Akshat
Quoted 14 lines
Quoted 6 lines
My problem with bonnie happens in a test which counts how many files where created. It just call readdir in loop and counts how many entries were returned. As far as I understand the bonnie code, there were no files deleted
yet.Quoted 7 lines
I checked the glibc code (i hope that the correct one) and it seems that the
loop inside readdir is returned only if there are no more data in the buffer
and so the getdents is called again. My problem is that there SHOULD be the last entry still in the buffer.
Quoted 4 lines
So, what you're saying is that the
getdents()call actually returns the last entry, butreaddir()somehow ignores that last entry? If that is the case, I would look at thed_offvalue for that last entry for weirdness.
Finally, we found that bug. Thank you all. The problem was that filp->f_pos was set to a wrong value when EOF was detected. And this value is assigned to ->d_off of the last returned entry after returning from vfs_readdir(). This value is checked by getdents() in glibc when checking if the kernel structures are the same as in userspace. The wrong value was 64bit (I forgot to truncate it to 32bits) whereas the userspace value is 32bit only :(
Tomas