New: ftw doesn't work like documented (may be a documentation bug)

The man page states:

int ftw(const char *dirpath,
int (*fn) (const char *fpath, const struct stat *sb,
int typeflag),
int nopenfd);

[...]

fpath is the pathname of the entry relative to dirpath.

Though it appears that it is the pathname of the entry relative to the current working directory.

FWIW this may be a documentation bug since the similar fts interface clearly states that the similar fts_accpath is relative to the current working directory.

Signature · 17 lines

           Summary: ftw doesn't work like documented (may be a documentation
                    bug)
           Product: glibc
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: libc
        AssignedTo: drepper at redhat dot com
        ReportedBy: madcoder at debian dot org
                CC: glibc-bugs at sources dot redhat dot com


http://sourceware.org/bugzilla/show_bug.cgi?id=11459

------- You are receiving this mail because: -------
You reported the bug, or are watching the reporter.

------- Additional Comments From drepper at redhat dot com 2010-04-03 20:11 ------- The man pages do not come with glibc and are in no way authoritative. You have to report this elsewhere.

Signature · 10 lines

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID


http://sourceware.org/bugzilla/show_bug.cgi?id=11459

------- You are receiving this mail because: -------
You reported the bug, or are watching the reporter.

See below a bug reported against the glibc.  Since the glibc maintainer dodged that one, I assume the bug indeed is in the documentation of ftw(3). My manpages are the 3.24-1 Debian package.

IMHO the patch is:

 -fpath is the pathname of the entry relative to dirpath.
 +fpath is the pathname of the entry relative to the current working directory.

POSIX is very vague about what "fpath" should be btw.

Quoted 36 lines

The man page states:

       int ftw(const char *dirpath,
               int (*fn) (const char *fpath, const struct stat *sb, int typeflag),
               int nopenfd);

       [...]

       fpath is the  pathname  of  the  entry  relative to dirpath.

Though it appears that it is the pathname of the entry relative to the current working directory.

FWIW this may be a documentation bug since the similar fts interface clearly states that the similar fts_accpath is relative to the current working directory.

--
           Summary: ftw doesn't work like documented (may be a documentation bug)
           Product: glibc
           Version: unspecified
            Status: NEW
          Severity: normal
          Priority: P2
         Component: libc
        AssignedTo: drepper at redhat dot com
        ReportedBy: madcoder at debian dot org
                CC: glibc-bugs at sources dot redhat dot com

http://sourceware.org/bugzilla/show_bug.cgi?id=11459

------- You are receiving this mail because: ------- You reported the bug, or are watching the reporter.

Quoted 12 lines · drepper at redhat dot com

------- Additional Comments From drepper at redhat dot com  2010-04-03 20:11 ------- The man pages do not come with glibc and are in no way authoritative.  You have to report this elsewhere.

--
           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED Resolution|                            |INVALID

Quoted 4 lines

http://sourceware.org/bugzilla/show_bug.cgi?id=11459

------- You are receiving this mail because: ------- You reported the bug, or are watching the reporter.

--
·O·  Pierre Habouzit
··O                                                [email protected] OOO                                                http://www.madism.org

Hello Pierre,

Quoted 3 lines · Pierre Habouzit

See below a bug reported against the glibc.  Since the glibc maintainer dodged that one, I assume the bug indeed is in the documentation of ftw(3). My manpages are the 3.24-1 Debian package.

Yes. The man page is clearly incorrect. Thanks for reporting this.

Quoted 6 lines

IMHO the patch is:

 -fpath is the pathname of the entry relative to dirpath.
 +fpath is the pathname of the entry relative to the current working directory.

POSIX is very vague about what "fpath" should be btw.

(Agreed. It could be more precise.)

I believe the correct text should be this:

fpath is the pathname of the entry, and is
expressed either as a pathname relative to the calling process's current working directory at the time of the call to ftw(), if dirpath was expressed as a relative pathname, or as an absolute pathname, if dirpath was expressed as an absolute pathname.

I have updated the man page accordingly, but would welcome review/checking of this text.

Quoted 27 lines · Michael Kerrisk

Hello Pierre,

Quoted 3 lines · Pierre Habouzit

See below a bug reported against the glibc.  Since the glibc maintainer dodged that one, I assume the bug indeed is in the documentation of ftw(3). My manpages are the 3.24-1 Debian package.

Yes. The man page is clearly incorrect. Thanks for reporting this.

Quoted 6 lines

IMHO the patch is:

 -fpath is the pathname of the entry relative to dirpath.
 +fpath is the pathname of the entry relative to the current working directory.

POSIX is very vague about what "fpath" should be btw.

(Agreed. It could be more precise.)

I believe the correct text should be this:

       fpath   is  the  pathname  of  the  entry,  and  is expressed either as  a  pathname  relative  to  the calling  process's current working directory at the time of the call to ftw(), if dirpath was expressed as a relative pathname, or as an absolute pathname, if dirpath was expressed as an  absolute  pathname.

Quoted 2 lines

I have updated the man page accordingly, but would welcome review/checking of this text.

Afaict, it's not correct: ftw may perform chdir() calls, so the pathname is relative to the current working directory at the time fn is called.

I'd rather phrase it that way (minus probable english mistakes):

   fpath is the pathname of the entry, and is either a relative pathname to the current working directory of the application when fn is called, or as an absolute pathname.

Hi Pierre,

Quoted 36 lines · Pierre Habouzit

Quoted 27 lines · Michael Kerrisk

Hello Pierre,

Quoted 3 lines · Pierre Habouzit

See below a bug reported against the glibc.  Since the glibc maintainer dodged that one, I assume the bug indeed is in the documentation of ftw(3). My manpages are the 3.24-1 Debian package.

Yes. The man page is clearly incorrect. Thanks for reporting this.

Quoted 6 lines

IMHO the patch is:

 -fpath is the pathname of the entry relative to dirpath.
 +fpath is the pathname of the entry relative to the current working directory.

POSIX is very vague about what "fpath" should be btw.

(Agreed. It could be more precise.)

I believe the correct text should be this:

       fpath   is  the  pathname  of  the  entry,  and  is expressed either as  a  pathname  relative  to  the calling  process's current working directory at the time of the call to ftw(), if dirpath was expressed as a relative pathname, or as an absolute pathname, if dirpath was expressed as an  absolute  pathname.

Quoted 2 lines

I have updated the man page accordingly, but would welcome review/checking of this text.

Afaict, it's not correct: ftw may perform chdir() calls, so the pathname is relative to the current working directory at the time fn is called.

I'd rather phrase it that way (minus probable english mistakes):

Quoted 3 lines

   fpath is the pathname of the entry, and is either a relative pathname to the current working directory of the application when fn is called, or as an absolute pathname.

Thanks for taking a look at this. However, I *think* your analysis is wrong, and my proposed changes is right. But, still I'd like some further confirmation. Please take a look at the the program below, and the output produced when it runs.

Quoted 43 lines · Michael Kerrisk

Hi Pierre,

Quoted 39 lines · Pierre Habouzit

Quoted 27 lines · Michael Kerrisk

Hello Pierre,

Quoted 3 lines · Pierre Habouzit

See below a bug reported against the glibc.  Since the glibc maintainer dodged that one, I assume the bug indeed is in the documentation of ftw(3). My manpages are the 3.24-1 Debian package.

Yes. The man page is clearly incorrect. Thanks for reporting this.

Quoted 6 lines

IMHO the patch is:

 -fpath is the pathname of the entry relative to dirpath.
 +fpath is the pathname of the entry relative to the current working directory.

POSIX is very vague about what "fpath" should be btw.

(Agreed. It could be more precise.)

I believe the correct text should be this:

       fpath   is  the  pathname  of  the  entry,  and  is expressed either as  a  pathname  relative  to  the calling  process's current working directory at the time of the call to ftw(), if dirpath was expressed as a relative pathname, or as an absolute pathname, if dirpath was expressed as an  absolute  pathname.

Quoted 2 lines

I have updated the man page accordingly, but would welcome review/checking of this text.

Afaict, it's not correct: ftw may perform chdir() calls, so the pathname is relative to the current working directory at the time fn is called.

I'd rather phrase it that way (minus probable english mistakes):

   fpath is the pathname of the entry, and is either a relative pathname to the current working directory of the application when fn is called, or as an absolute pathname.

Quoted 4 lines

Thanks for taking a look at this. However, I *think* your analysis is wrong, and my proposed changes is right. But, still I'd like some further confirmation. Please take a look at the the program below, and the output produced when it runs.

Yeah, that's because you're doing chdir()s during ftw, which is undefined behaviour as documented in the manpage already IIRC.

The point is, ftw() /may/ decide to do chdir() by itself sometimes, and then the path is relative to the current working directory as set by ftw().

I'm pretty sure it's what POSIX authorizes ftw() to perform chdirs.

And when I look at ftw.c in the glibc, it's also pretty much what happens in the case when you set FTW_CHDIR in the flags: ftw() forces a chdir before the fn() call, and makes the path relative to this cwd.

It happens that the glibc doesn't seem to perform any kind of chdir() in the other cases (IOW when FTW_CHDIR isn't set), but I'm pretty sure POSIX allows ftw() to do so.

Hi Pierre,

Thanks for following up on this. I still think the current man page text is correct. See below.

Quoted 44 lines · Pierre Habouzit

Quoted 43 lines · Michael Kerrisk

Hi Pierre,

Quoted 39 lines · Pierre Habouzit

Quoted 27 lines · Michael Kerrisk

Hello Pierre,

Quoted 3 lines · Pierre Habouzit

See below a bug reported against the glibc.  Since the glibc maintainer dodged that one, I assume the bug indeed is in the documentation of ftw(3). My manpages are the 3.24-1 Debian package.

Yes. The man page is clearly incorrect. Thanks for reporting this.

Quoted 6 lines

IMHO the patch is:

 -fpath is the pathname of the entry relative to dirpath.
 +fpath is the pathname of the entry relative to the current working directory.

POSIX is very vague about what "fpath" should be btw.

(Agreed. It could be more precise.)

I believe the correct text should be this:

       fpath   is  the  pathname  of  the  entry,  and  is expressed either as  a  pathname  relative  to  the calling  process's current working directory at the time of the call to ftw(), if dirpath was expressed as a relative pathname, or as an absolute pathname, if dirpath was expressed as an  absolute  pathname..

Quoted 2 lines

I have updated the man page accordingly, but would welcome review/checking of this text.

Afaict, it's not correct: ftw may perform chdir() calls, so the pathname is relative to the current working directory at the time fn is called.

I'd rather phrase it that way (minus probable english mistakes):

   fpath is the pathname of the entry, and is either a relative pathname to the current working directory of the application when fn is called, or as an absolute pathname.

Quoted 7 lines

Quoted 4 lines

Thanks for taking a look at this. However, I *think* your analysis is wrong, and my proposed changes is right. But, still I'd like some further confirmation. Please take a look at the the program below, and the output produced when it runs.

Yeah, that's because you're doing chdir()s during ftw, which is undefined behaviour as documented in the manpage already IIRC.

True. That is documented in the POSIX page, but not currently in man-pages. I've fixed that now. Thanks.

Quoted 3 lines

The point is, ftw() /may/ decide to do chdir() by itself sometimes, and then the path is relative to the current working directory as set by ftw().

I'm not sure why you think it may decide to do this. If this was unpredictable, then it would create difficulties for the application, as far as I can tell, since it would take quite some effort to correctly interpret the pathname supplied to 'fn'. And POSIX seems fairly clear on the point (at least for nftw()):

FTW_CHDIR
If set, nftw() shall change the current working directory to each directory as it reports files in that directory. If clear, nftw() shall not change the current working directory.

Quoted 1 line

I'm pretty sure it's what POSIX authorizes ftw() to perform chdirs.

See above.

Quoted 3 lines

And when I look at ftw.c in the glibc, it's also pretty much what happens in the case when you set FTW_CHDIR in the flags: ftw() forces a chdir before the fn() call, and makes the path relative to this cwd.

Yes to the first part, but as far as I can see, still no to the last part ("and makes the path relative to this cwd" ). See below.

Quoted 3 lines

It happens that the glibc doesn't seem to perform any kind of chdir() in the other cases (IOW when FTW_CHDIR isn't set), but I'm pretty sure POSIX allows ftw() to do so.

I don't see anywhere that POSIX authorizes that, and it wouldn't seem sensible to do so. See above.

Here's a variation on my earlier test program that uses nftw() instead. If "c" is provided in the second command line argument, it uses "FTW_CHDIR". Looking at the results, do you agree with my analysis?

$ cat n.c
#define _GNU_SOURCE
#define _XOPEN_SOURCE 500
#include <ftw.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

static int
displayFileInfo(const char *fpath, const struct stat *sb,
int tflag, struct FTW *ftwbuf)

{
    printf("%-3s %2d %7lld   %-40s %d %s\n",
        (tflag == FTW_D) ?   "d"   : (tflag == FTW_DNR) ? "dnr" :
        (tflag == FTW_DP) ?  "dp"  : (tflag == FTW_F) ?   "f" :
        (tflag == FTW_NS) ?  "ns"  : (tflag == FTW_SL) ?  "sl" :
        (tflag == FTW_SLN) ? "sln" : "???",
        ftwbuf->level, (long long) sb->st_size,
        fpath, ftwbuf->base, fpath + ftwbuf->base);

#ifdef DO_CHDIR
/* Let's mess with the curent directory during the ftw() call,
to see what value is passed to 'pathname' in successive calls to displayFileInfo() */

chdir("..");
#endif
system("pwd");
return 0; /* To tell nftw() to continue */
}

int
main(int argc, char *argv[])

{
    int flags = 0;

    if (argc > 2 && strchr(argv[2], 'd') != NULL)
        flags |= FTW_DEPTH;
    if (argc > 2 && strchr(argv[2], 'p') != NULL)
        flags |= FTW_PHYS;
    if (argc > 2 && strchr(argv[2], 'c') != NULL)
        flags |= FTW_CHDIR;

    if (nftw((argc < 2) ? "." : argv[1], displayFileInfo, 20, flags) == -1) {
        perror("nftw");
        exit(EXIT_FAILURE);
    }

    exit(EXIT_SUCCESS);
}

$ cc -o n n.c
$ cd dir1
/home/mtk/tlpi/dl/dir1
$ find ../dir2
.../dir2
.../dir2/sub
.../dir2/sub/d
.../dir2/sub/b
.../dir2/sub/c
.../dir2/sub/a
.../dir2/bbb
$ ../n ../dir2 c
d    0    4096   ../dir2                                  3 dir2
/home/mtk/tlpi/dirs_links
d    1    4096   ../dir2/sub                              8 sub
/home/mtk/tlpi/dirs_links/dir2
f    2       0   ../dir2/sub/d                            12 d
/home/mtk/tlpi/dirs_links/dir2/sub
f    2       0   ../dir2/sub/b                            12 b
/home/mtk/tlpi/dirs_links/dir2/sub
f    2       0   ../dir2/sub/c                            12 c
/home/mtk/tlpi/dirs_links/dir2/sub
f    2       0   ../dir2/sub/a                            12 a
/home/mtk/tlpi/dirs_links/dir2/sub
f    1      38   ../dir2/bbb                              8 bbb
/home/mtk/tlpi/dirs_links/dir2
$ ../n ../dir2
d    0    4096   ../dir2                                  3 dir2
/home/mtk/tlpi/dl/dir1
d    1    4096   ../dir2/sub                              8 sub
/home/mtk/tlpi/dl/dir1
f    2       0   ../dir2/sub/d                            12 d
/home/mtk/tlpi/dl/dir1
f    2       0   ../dir2/sub/b                            12 b
/home/mtk/tlpi/dl/dir1
f    2       0   ../dir2/sub/c                            12 c
/home/mtk/tlpi/dl/dir1
f    2       0   ../dir2/sub/a                            12 a
/home/mtk/tlpi/dl/dir1
f    1      38   ../dir2/bbb                              8 bbb
/home/mtk/tlpi/dl/dir1

Thanks,

Michael

I think it makes sense indeed.

Quoted 193 lines

Hi Pierre,

Thanks for following up on this. I still think the current man page text is correct. See below.

Quoted 44 lines · Pierre Habouzit

Quoted 43 lines · Michael Kerrisk

Hi Pierre,

Quoted 39 lines · Pierre Habouzit

Quoted 27 lines · Michael Kerrisk

Hello Pierre,

Quoted 3 lines · Pierre Habouzit

See below a bug reported against the glibc.  Since the glibc maintainer dodged that one, I assume the bug indeed is in the documentation of ftw(3). My manpages are the 3.24-1 Debian package.

Yes. The man page is clearly incorrect. Thanks for reporting this.

Quoted 6 lines

IMHO the patch is:

 -fpath is the pathname of the entry relative to dirpath.
 +fpath is the pathname of the entry relative to the current working directory.

POSIX is very vague about what "fpath" should be btw.

(Agreed. It could be more precise.)

I believe the correct text should be this:

       fpath   is  the  pathname  of  the  entry,  and  is expressed either as  a  pathname  relative  to  the calling  process's current working directory at the time of the call to ftw(), if dirpath was expressed as a relative pathname, or as an absolute pathname, if dirpath was expressed as an  absolute  pathname..

Quoted 2 lines

I have updated the man page accordingly, but would welcome review/checking of this text.

Afaict, it's not correct: ftw may perform chdir() calls, so the pathname is relative to the current working directory at the time fn is called.

I'd rather phrase it that way (minus probable english mistakes):

   fpath is the pathname of the entry, and is either a relative pathname to the current working directory of the application when fn is called, or as an absolute pathname.

Quoted 7 lines

Quoted 4 lines

Thanks for taking a look at this. However, I *think* your analysis is wrong, and my proposed changes is right. But, still I'd like some further confirmation. Please take a look at the the program below, and the output produced when it runs.

Yeah, that's because you're doing chdir()s during ftw, which is undefined behaviour as documented in the manpage already IIRC.

True. That is documented in the POSIX page, but not currently in man-pages. I've fixed that now. Thanks.

Quoted 3 lines

The point is, ftw() /may/ decide to do chdir() by itself sometimes, and then the path is relative to the current working directory as set by ftw().

I'm not sure why you think it may decide to do this. If this was unpredictable, then it would create difficulties for the application, as far as I can tell, since it would take quite some effort to correctly interpret the pathname supplied to 'fn'. And POSIX seems fairly clear on the point (at least for nftw()):

       FTW_CHDIR
              If set, nftw() shall change  the  current  working directory to each directory as it reports files in that directory. If clear, nftw() shall not  change the current working directory.

Quoted 1 line

I'm pretty sure it's what POSIX authorizes ftw() to perform chdirs.

See above.

Quoted 3 lines

And when I look at ftw.c in the glibc, it's also pretty much what happens in the case when you set FTW_CHDIR in the flags: ftw() forces a chdir before the fn() call, and makes the path relative to this cwd.

Yes to the first part, but as far as I can see, still no to the last part ("and makes the path relative to this cwd" ). See below.

Quoted 3 lines

It happens that the glibc doesn't seem to perform any kind of chdir() in the other cases (IOW when FTW_CHDIR isn't set), but I'm pretty sure POSIX allows ftw() to do so.

I don't see anywhere that POSIX authorizes that, and it wouldn't seem sensible to do so. See above.

Here's a variation on my earlier test program that uses nftw() instead. If "c" is provided in the second command line argument, it uses "FTW_CHDIR". Looking at the results, do you agree with my analysis?

$ cat n.c
#define _GNU_SOURCE
#define _XOPEN_SOURCE 500
#include <ftw.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

static int
displayFileInfo(const char *fpath, const struct stat *sb, int tflag, struct FTW *ftwbuf)
{
    printf("%-3s %2d %7lld   %-40s %d %s\n",
        (tflag == FTW_D) ?   "d"   : (tflag == FTW_DNR) ? "dnr" :
        (tflag == FTW_DP) ?  "dp"  : (tflag == FTW_F) ?   "f" :
        (tflag == FTW_NS) ?  "ns"  : (tflag == FTW_SL) ?  "sl" :
        (tflag == FTW_SLN) ? "sln" : "???", ftwbuf->level, (long long) sb->st_size,
        fpath, ftwbuf->base, fpath + ftwbuf->base); #ifdef DO_CHDIR
    /* Let's mess with the curent directory during the ftw() call, to see what value is passed to 'pathname' in successive calls to displayFileInfo() */

    chdir("..");
#endif
    system("pwd");
    return 0;           /* To tell nftw() to continue */ }

int
main(int argc, char *argv[])
{
    int flags = 0;

    if (argc > 2 && strchr(argv[2], 'd') != NULL)
        flags |= FTW_DEPTH;
    if (argc > 2 && strchr(argv[2], 'p') != NULL)
        flags |= FTW_PHYS;
    if (argc > 2 && strchr(argv[2], 'c') != NULL)
        flags |= FTW_CHDIR;

    if (nftw((argc < 2) ? "." : argv[1], displayFileInfo, 20, flags) == -1) { perror("nftw");
        exit(EXIT_FAILURE);
    }

    exit(EXIT_SUCCESS);
}

$ cc -o n n.c
$ cd dir1
/home/mtk/tlpi/dl/dir1
$ find ../dir2
.../dir2
.../dir2/sub
.../dir2/sub/d
.../dir2/sub/b
.../dir2/sub/c
.../dir2/sub/a
.../dir2/bbb
$ ../n ../dir2 c
d    0    4096   ../dir2                                  3 dir2
/home/mtk/tlpi/dirs_links
d    1    4096   ../dir2/sub                              8 sub
/home/mtk/tlpi/dirs_links/dir2
f    2       0   ../dir2/sub/d                            12 d
/home/mtk/tlpi/dirs_links/dir2/sub
f    2       0   ../dir2/sub/b                            12 b
/home/mtk/tlpi/dirs_links/dir2/sub
f    2       0   ../dir2/sub/c                            12 c
/home/mtk/tlpi/dirs_links/dir2/sub
f    2       0   ../dir2/sub/a                            12 a
/home/mtk/tlpi/dirs_links/dir2/sub
f    1      38   ../dir2/bbb                              8 bbb
/home/mtk/tlpi/dirs_links/dir2
$ ../n ../dir2
d    0    4096   ../dir2                                  3 dir2
/home/mtk/tlpi/dl/dir1
d    1    4096   ../dir2/sub                              8 sub
/home/mtk/tlpi/dl/dir1
f    2       0   ../dir2/sub/d                            12 d
/home/mtk/tlpi/dl/dir1
f    2       0   ../dir2/sub/b                            12 b
/home/mtk/tlpi/dl/dir1
f    2       0   ../dir2/sub/c                            12 c
/home/mtk/tlpi/dl/dir1
f    2       0   ../dir2/sub/a                            12 a
/home/mtk/tlpi/dl/dir1
f    1      38   ../dir2/bbb                              8 bbb
/home/mtk/tlpi/dl/dir1

Quoted 3 lines

Thanks,

Michael

--
·O·  Pierre Habouzit
··O                                                [email protected] OOO                                                http://www.madism.org