Skip to contentSkip to conversation

nfs question - ftruncate vs pwrite

DISCUSSION Kenny Simpson 3 participants 6 messages Dec 7, 2005 – Dec 8, 2005

lore
Share & tools
lore
#0 lore raw

Sorry about the previous partial message...

If a file is extended via ftruncate, the new empty pages are read in before the the ftruncate returns (taking 64mS on my machine), but if the file is extended via pwrite, nothing is read in and the system call is very quick (34uS).

Why is there such a difference? Is there another cheap way to grow a file and map in its new pages? Am I missing some other semantic difference between ftruncate and a pwrite past the end of the file?

Here is a test program.. compile with -DABUSE to get the pwrite version.

thanks,
-Kenny

__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com

#define _GNU_SOURCE

#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

#include <stdio.h>
#include <string.h>
#include <malloc.h>

int main(int argc, char* argv[])
{
int fd;
unsigned long long int const size = 4096 * 1024; unsigned int const size_page = 1024; unsigned long long int offset = 0; unsigned int offset_page = 0;

  //char* buffer = valloc(size);
  //memset(buffer, 0, size);

  if (argc != 2) {
    printf("usage: %s <filename>\n", argv[0]);
    return 0;
  }

  fd = open(argv[1], O_RDWR | O_CREAT | O_TRUNC | O_LARGEFILE /*| O_DIRECT*/, 0644);
  if (fd < 0) {
    perror("open");
    return 0;
  }

#ifdef ABUSE
pwrite64(fd, "" , 1, offset + size);
#else
ftruncate64(fd, offset + size);
#endif

  char* mapping = (char*)mmap64(0, size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE | MAP_NONBLOCK, fd, offset);
  memset(mapping, 'a', size);

  for (;;) {
    offset += size;
    offset_page += size_page;
#ifdef ABUSE
    pwrite64(fd, "", 1, offset + size);
#else
    ftruncate64(fd, offset + size);

#endif

    //munmap(mapping, size);
    //mapping = (char*)mmap64(0, size, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE | MAP_NONBLOCK, fd, offset);
    remap_file_pages(mapping, size, 0, offset_page, MAP_NONBLOCK);

    memset(mapping, 'a', size);

}

  close(fd);

return 0;
}

#1 lore raw
Quoted 10 lines · Kenny Simpson

Sorry about the previous partial message...

If a file is extended via ftruncate, the new empty pages are read in before the the ftruncate returns (taking 64mS on my machine), but if the file is extended via pwrite, nothing is read in and the system call is very quick (34uS).

Why is there such a difference? Is there another cheap way to grow a file and map in its new pages? Am I missing some other semantic difference between ftruncate and a pwrite past the end of the file?

You might use tcpdump or etherreal to see what the different traffic looks like. I suspect that ftruncate() leads a SETATTR operation while pwrite() leads to a WRITE operation.

ps

#2 lore raw
Quoted 3 lines · --- Peter Staubach

You might use tcpdump or etherreal to see what the different traffic looks like. I suspect that ftruncate() leads a SETATTR operation while pwrite() leads to a WRITE operation.

Ethereal results interpreted with wild speculation:
The pwrite case:
This does a bunch of reads, but the server always returns a short read responding with EOF. It
seems that a pwrite does cause a getattr call, but that's it.
Once memory is exhausted, the pages are written out.

The ftruncate case:
This does a setattr, then does a read - this time the server responds with a large amount of
0's.

Since this is using the buffer cache (not opened with O_DIRECT), and since we know we are extending the file... is it strictly necessary to read in pages of 0's from the server?

-Kenny

__________________________________________
Yahoo! DSL – Something to write home about.
Just $16.99/mo. or less.
dsl.yahoo.com

#3 lore raw
Quoted 11 lines · Kenny Simpson
Quoted 3 lines · --- Peter Staubach

You might use tcpdump or etherreal to see what the different traffic looks like. I suspect that ftruncate() leads a SETATTR operation while pwrite() leads to a WRITE operation.

Ethereal results interpreted with wild speculation:
The pwrite case:
This does a bunch of reads, but the server always returns a short read responding with EOF. It
seems that a pwrite does cause a getattr call, but that's it.
Once memory is exhausted, the pages are written out.

Quoted 3 lines

The ftruncate case:
This does a setattr, then does a read - this time the server responds with a large amount of
0's.

That is as expected. The ftruncate() causes an immediate change in length of the file on the server, and so reads will. In the case of pwrite(), that is cached on the client until you fsync/close, and so the server returns short reads.

Quoted 2 lines

Since this is using the buffer cache (not opened with O_DIRECT), and since we know we are extending the file... is it strictly necessary to read in pages of 0's from the server?

Possibly not, but is this a common case that is worth optimising for? Note that use of the standard write() syscall as opposed to mmap() will not trigger this avalanche of page-ins.

Cheers,
Trond

#4 lore raw
Quoted 12 lines · Trond Myklebust
Quoted 11 lines · Kenny Simpson
Quoted 3 lines · --- Peter Staubach

You might use tcpdump or etherreal to see what the different traffic looks like. I suspect that ftruncate() leads a SETATTR operation while pwrite() leads to a WRITE operation.

Ethereal results interpreted with wild speculation:
The pwrite case:
This does a bunch of reads, but the server always returns a short read responding with EOF. It
seems that a pwrite does cause a getattr call, but that's it.
Once memory is exhausted, the pages are written out.

Quoted 6 lines
Quoted 3 lines

The ftruncate case:
This does a setattr, then does a read - this time the server responds with a large amount of
0's.

That is as expected. The ftruncate() causes an immediate change in length of the file on the server, and so reads will.

...Err...

...and so reads of the empty pages will succeed.

Quoted 19 lines

In the case of
pwrite(), that is cached on the client until you fsync/close, and so the server returns short reads.

Quoted 2 lines

Since this is using the buffer cache (not opened with O_DIRECT), and since we know we are extending the file... is it strictly necessary to read in pages of 0's from the server?

Possibly not, but is this a common case that is worth optimising for? Note that use of the standard write() syscall as opposed to mmap() will not trigger this avalanche of page-ins.

Cheers,
Trond

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#5 lore raw
In reply to Trond Myklebust
Quoted 9 lines · --- Trond Myklebust

That is as expected. The ftruncate() causes an immediate change in length of the file on the server, and so reads will. In the case of pwrite(), that is cached on the client until you fsync/close, and so the server returns short reads.

Quoted 2 lines

Since this is using the buffer cache (not opened with O_DIRECT), and since we know we are extending the file... is it strictly necessary to read in pages of 0's from the server?

Possibly not, but is this a common case that is worth optimising for?

I am attempting to write a low-latency logger. 'Low' meaning a system call is too slow (measured at 0.3 microseconds) for each message. So I am trying to use the page cache to handle the background scheduling of bulk writes to the server, and as an extra layer of reliability in the event of a program crash. The use of pwrite seems to be the best option at this time as spending a few milliseconds for an ftruncate to a show-stopper.

I could also just write locally into a shared memory region, and have my own background copy to the server, but this seems a bit wasteful when the page cache does most of this already, and can optimize page-sized writes.

-Kenny

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com