Skip to contentSkip to conversation

NFS file size anomaly?

DISCUSSION Dan Duval 2 participants 2 messages Dec 11, 2013

Subsystems (3)
lore
Share & tools
lore

Dan Duval

author
#0 lore raw
Quoted 193 lines · ----- [email protected] wrote:
Quoted 3 lines · Trond Myklebust
Quoted 2 lines · Jan Kara
Quoted 1 line · Chuck Lever

On Dec 9, 2013, at 4:05 PM, Chuck Lever <[email protected]>

Quoted 9 lines · wrote:
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Copying Trond's new work address.

Begin forwarded message:

Quoted 5 lines

From: Dan Duval <[email protected]>
Subject: NFS file size anomaly?
Date: December 9, 2013 3:04:27 PM EST
To: Trond Myklebust <[email protected]>
Cc: <[email protected]>,

<[email protected]>

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

[NOTE: cross-posted to linux-nfs and linux-fsdevel]

I'm seeing some unexpected behavior with NFS and file sizes.

The test cases are from the LTP (Linux Test Project), tests ftest01, ftest05, and ftest07. I'll concentrate on ftest01 to explain what I've found.

ftest01 fires off 5 subprocesses, each of which opens an

empty

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

file and does the following, repeatedly:

. lseek to some point in the file
. read 2048 bytes
. lseek back to the same point
. write 2048 bytes

The "point in the file" is determined by a pseudo-random sequence. All such points are on 2048-byte boundaries.

Occasionally, also driven pseudo-randomly, ftest01 will

throw

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

in a call to ftruncate(), truncate(), sync(), or fstat().

With the fstat() calls, the returned .st_size is compared with the test's expected size for the file, and an error is declared if they don't match.

What's happening is that, some way into the test, this

fstat()

Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines

check is failing. Specifically, the .st_size reported by fstat() is greater than the computed size.

The sequence of operations leading up to this is:

      lseek 1034240 0
      read 2048
      lseek 0 1
      write 2048

      lseek 638976 0
      (read, lseek, write)

      lseek 708608 0
      (read, lseek, write)

      lseek 708608 0
      (read, lseek, write)

      lseek 679584 0
      (read, lseek, write)

      truncate 266240

      lseek 960512 0
      (read, lseek, write)

      (a bunch of lseek/read/lseek/write ops that do not
       extend the file)

      fstat
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

So the expected size of the file is 960512 + 2048 == 960560. But the fstat reports a size of 1036288.

A look at what's happening on the wire, distilled from the output of tethereal, is instructive.

READ Call 638976 4096 (byte offset and size to read) READ Reply 4096 995382 (bytes read and current file

size)

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

SETATTR Call 266240 (this corresponds to the truncate()

call)

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
      WRITE Call 638976 4096 (byte offset and size to write)
      WRITE Call 708608 4096
      WRITE Call 1032192 4096

      SETATTR Reply 266240 (current size of file)

      WRITE Reply 643072 (current size of file after write)
      WRITE Reply 1036288
      WRITE Reply 1036288
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

GETATTR (initiated internally by NFS code?)

READ Call 958464 4096 READ Reply 4096 1036288

... (a bunch of READ and WRITE ops that do not extend

the file)

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

GETATTR Call (this corresponds to the fstat() call) GETATTR Reply 1036288

So what appears to have happened here is that three of the WRITE operations that the program issued before the

truncate()

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

call have "bled past" the SETATTR, extending the file

further

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

than the SETATTR did. Since none of the operations issued after SETATTR extends the file further, by the time we get

to

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

the GETATTR, the file is larger than the test expects.

There are two strange things going on here. The first, identified above, is that write()s that were initiated

before

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

the truncate() call are being processed after the resulting SETATTR call. The second is that WRITE operations are being initiated while the SETATTR is outstanding.

It seems to me that a size-changing SETATTR operation should act essentially as an I/O barrier. It should wait for all

outstanding

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

read/write requests to complete, then issue the SETATTR, wait for the reply, and only then re-enable read/write

requests.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

In other words, SETATTR should be atomic with respect to

other

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

I/O operations.

nfs_setattr() relies on writeback_single_inode() [ via

nfs_setattr() ->

Quoted 1 line
Quoted 1 line
Quoted 1 line

nfs_wb_all() -> sync_inode() -> writeback_single_inode() ] to

serialize

Quoted 1 line
Quoted 1 line
Quoted 1 line

itself with other pending I/O operations. The NFS client

already treats

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

SETATTR as an I/O barrier.

Quoted 5 lines
Quoted 5 lines

A git bisect indicates that this problem first appeared (or was first uncovered) with this commit:

4f8ad65 writeback: Refactor writeback_single_inode()

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

It continues to the most recent mainline kernels.

NFS v3 vs. v4 doesn't seem to matter.

Has anyone else seen this? Any pointers you can provide?

After commit 4f8ad65, writeback_single_inode() occasionally

returns

Quoted 1 line
Quoted 1 line
Quoted 1 line

immediately instead of waiting for writeback to finish. The

behavior

Quoted 1 line
Quoted 1 line
Quoted 1 line

would be difficult or impossible to spot with a local

filesystem, but

Quoted 2 lines
Quoted 2 lines
Quoted 1 line

with NFS, a race is clearly exposed.

That looks like a bug. But the only thing that's changed by

4f8ad65 is

Quoted 4 lines
Quoted 4 lines

the additional test:

	if (!(inode->i_state & I_DIRTY))
		goto out;

So it seems NFS has inodes that have dirty data but don't have

(any of) inode

Quoted 1 line
Quoted 1 line

dirty bits set? But I've been staring at the code and don't see

any

Quoted 1 line
Quoted 1 line

problem. Any idea Chuck? Otherwise I guess I'll have to reproduce

this and

Quoted 6 lines
Quoted 1 line

debug what's going on.

Hi Jan,

I'm guessing that the problem is the fact that you can clear the I_DIRTY_PAGES flag inside __writeback_single_inode() on a

non-waiting

Quoted 2 lines

call (i.e. one which does not have WB_SYNC_ALL set). When that happens, the NFS client ends up with inode->i_state == 0

until

Quoted 1 line

one of the WRITEs returns with the "unstable" flag set (which

triggers a

Quoted 4 lines

call to __mark_inode_dirty(inode, I_DIRTY_DATASYNC) so that we call write_inode()).

Basically, if you have WB_SYNC_ALL set, I think that you cannot

apply

Quoted 11 lines
Quoted 2 lines

the test for (inode->i_state & I_DIRTY) until you have called filemap_fdatawait().

Ah, thanks for explanation. I should have spotted this myself. So
something like the attached patch should fix the issue. Dan, can you try
whether the attached patch fixes your problem?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR

Jan,

Confirmed - this patch eliminates the errors I was seeing. The LTP tests now pass reliably.

Dan Duval <[email protected]>

#1 lore raw
Quoted 208 lines · Dan Duval
Quoted 193 lines · ----- [email protected] wrote:
Quoted 3 lines · Trond Myklebust
Quoted 2 lines · Jan Kara
Quoted 1 line · Chuck Lever

On Dec 9, 2013, at 4:05 PM, Chuck Lever <[email protected]>

Quoted 9 lines · wrote:
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

Copying Trond's new work address.

Begin forwarded message:

Quoted 5 lines

From: Dan Duval <[email protected]>
Subject: NFS file size anomaly?
Date: December 9, 2013 3:04:27 PM EST
To: Trond Myklebust <[email protected]>
Cc: <[email protected]>,

<[email protected]>

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

[NOTE: cross-posted to linux-nfs and linux-fsdevel]

I'm seeing some unexpected behavior with NFS and file sizes.

The test cases are from the LTP (Linux Test Project), tests ftest01, ftest05, and ftest07. I'll concentrate on ftest01 to explain what I've found.

ftest01 fires off 5 subprocesses, each of which opens an

empty

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

file and does the following, repeatedly:

. lseek to some point in the file
. read 2048 bytes
. lseek back to the same point
. write 2048 bytes

The "point in the file" is determined by a pseudo-random sequence. All such points are on 2048-byte boundaries.

Occasionally, also driven pseudo-randomly, ftest01 will

throw

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

in a call to ftruncate(), truncate(), sync(), or fstat().

With the fstat() calls, the returned .st_size is compared with the test's expected size for the file, and an error is declared if they don't match.

What's happening is that, some way into the test, this

fstat()

Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines

check is failing. Specifically, the .st_size reported by fstat() is greater than the computed size.

The sequence of operations leading up to this is:

      lseek 1034240 0
      read 2048
      lseek 0 1
      write 2048

      lseek 638976 0
      (read, lseek, write)

      lseek 708608 0
      (read, lseek, write)

      lseek 708608 0
      (read, lseek, write)

      lseek 679584 0
      (read, lseek, write)

      truncate 266240

      lseek 960512 0
      (read, lseek, write)

      (a bunch of lseek/read/lseek/write ops that do not
       extend the file)

      fstat
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

So the expected size of the file is 960512 + 2048 == 960560. But the fstat reports a size of 1036288.

A look at what's happening on the wire, distilled from the output of tethereal, is instructive.

READ Call 638976 4096 (byte offset and size to read) READ Reply 4096 995382 (bytes read and current file

size)

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

SETATTR Call 266240 (this corresponds to the truncate()

call)

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
      WRITE Call 638976 4096 (byte offset and size to write)
      WRITE Call 708608 4096
      WRITE Call 1032192 4096

      SETATTR Reply 266240 (current size of file)

      WRITE Reply 643072 (current size of file after write)
      WRITE Reply 1036288
      WRITE Reply 1036288
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

GETATTR (initiated internally by NFS code?)

READ Call 958464 4096 READ Reply 4096 1036288

... (a bunch of READ and WRITE ops that do not extend

the file)

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

GETATTR Call (this corresponds to the fstat() call) GETATTR Reply 1036288

So what appears to have happened here is that three of the WRITE operations that the program issued before the

truncate()

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

call have "bled past" the SETATTR, extending the file

further

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

than the SETATTR did. Since none of the operations issued after SETATTR extends the file further, by the time we get

to

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

the GETATTR, the file is larger than the test expects.

There are two strange things going on here. The first, identified above, is that write()s that were initiated

before

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

the truncate() call are being processed after the resulting SETATTR call. The second is that WRITE operations are being initiated while the SETATTR is outstanding.

It seems to me that a size-changing SETATTR operation should act essentially as an I/O barrier. It should wait for all

outstanding

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

read/write requests to complete, then issue the SETATTR, wait for the reply, and only then re-enable read/write

requests.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

In other words, SETATTR should be atomic with respect to

other

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line

I/O operations.

nfs_setattr() relies on writeback_single_inode() [ via

nfs_setattr() ->

Quoted 1 line
Quoted 1 line
Quoted 1 line

nfs_wb_all() -> sync_inode() -> writeback_single_inode() ] to

serialize

Quoted 1 line
Quoted 1 line
Quoted 1 line

itself with other pending I/O operations. The NFS client

already treats

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

SETATTR as an I/O barrier.

Quoted 5 lines
Quoted 5 lines

A git bisect indicates that this problem first appeared (or was first uncovered) with this commit:

4f8ad65 writeback: Refactor writeback_single_inode()

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

It continues to the most recent mainline kernels.

NFS v3 vs. v4 doesn't seem to matter.

Has anyone else seen this? Any pointers you can provide?

After commit 4f8ad65, writeback_single_inode() occasionally

returns

Quoted 1 line
Quoted 1 line
Quoted 1 line

immediately instead of waiting for writeback to finish. The

behavior

Quoted 1 line
Quoted 1 line
Quoted 1 line

would be difficult or impossible to spot with a local

filesystem, but

Quoted 2 lines
Quoted 2 lines
Quoted 1 line

with NFS, a race is clearly exposed.

That looks like a bug. But the only thing that's changed by

4f8ad65 is

Quoted 4 lines
Quoted 4 lines

the additional test:

	if (!(inode->i_state & I_DIRTY))
		goto out;

So it seems NFS has inodes that have dirty data but don't have

(any of) inode

Quoted 1 line
Quoted 1 line

dirty bits set? But I've been staring at the code and don't see

any

Quoted 1 line
Quoted 1 line

problem. Any idea Chuck? Otherwise I guess I'll have to reproduce

this and

Quoted 6 lines
Quoted 1 line

debug what's going on.

Hi Jan,

I'm guessing that the problem is the fact that you can clear the I_DIRTY_PAGES flag inside __writeback_single_inode() on a

non-waiting

Quoted 2 lines

call (i.e. one which does not have WB_SYNC_ALL set). When that happens, the NFS client ends up with inode->i_state == 0

until

Quoted 1 line

one of the WRITEs returns with the "unstable" flag set (which

triggers a

Quoted 4 lines

call to __mark_inode_dirty(inode, I_DIRTY_DATASYNC) so that we call write_inode()).

Basically, if you have WB_SYNC_ALL set, I think that you cannot

apply

Quoted 11 lines
Quoted 2 lines

the test for (inode->i_state & I_DIRTY) until you have called filemap_fdatawait().

Ah, thanks for explanation. I should have spotted this myself. So
something like the attached patch should fix the issue. Dan, can you try
whether the attached patch fixes your problem?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
Quoted 4 lines

Jan,

Confirmed - this patch eliminates the errors I was seeing. The LTP tests now pass reliably.

Thanks for confirmation. I've submitted the patch for inclusion.

Honza

Signature · 6 lines
Jan Kara <[email protected]>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html