Skip to contentSkip to conversation

Samba speed

DISCUSSION Jeremy Allison 6 participants 10 messages Dec 8, 2008 – Dec 9, 2008 VFS

lore
Share & tools
lore
#1 lore raw
Quoted 8 lines · Jeremy Allison

Here's a really interesting paper from Intel that they recently brought to my attention.

http://software.intel.com/en-us/articles/windows-client-cifs-behavior-can-slow-linux-nas-performance

Looks like using XFS for your Linux Samba server, or setting "strict allocate = yes" can make a big difference due to sparse file issues.

Glibc 2.7 (as shipped in Ubuntu Hardy) has posix_fallocate wired up to the fallocate system call, and ext4 supports delayed allocation as well as preallocation. There are number of userspace applications --- rsync, samba, and most bittorrent applications come to mind --- where use of fallocate would be a big win.

- Ted

#2 lore raw
Quoted 10 lines · Theodore Tso
Quoted 8 lines · Jeremy Allison

Here's a really interesting paper from Intel that they recently brought to my attention.

http://software.intel.com/en-us/articles/windows-client-cifs-behavior-can-slow-linux-nas-performance

Looks like using XFS for your Linux Samba server, or setting "strict allocate = yes" can make a big difference due to sparse file issues.

Quoted 5 lines

Glibc 2.7 (as shipped in Ubuntu Hardy) has posix_fallocate wired up to the fallocate system call, and ext4 supports delayed allocation as well as preallocation. There are number of userspace applications --- rsync, samba, and most bittorrent applications come to mind --- where use of fallocate would be a big win.

Turns out that ext4 doesn't suffer from the slowdown in the first place. The paper is extremly interesting, I'm looking at the implications for our default settings (most users are still using Samba on ext3 on Linux).

Jeremy.

#3 lore raw
Quoted 5 lines · Jeremy Allison

Turns out that ext4 doesn't suffer from the slowdown in the first place. The paper is extremly interesting, I'm looking at the implications for our default settings (most users are still using Samba on ext3 on Linux).

I thought the paper only talked about ext3, and theorized that delayed allocation in ext4 might be enough to make the problem go away, but they had not actually done any measurements to confirm this supposition. Has there been any more recent benchmarks comparing ext3, ext4, and XFS running Samba serving Windows clients?

- Ted

P.S. I'll be on the Google campus tomorrow and Wednesday attending the Ubuntu developer's conference; we should get together for lunch or dinner or some such....

#4 lore raw
Quoted 7 lines · Theodore Ts'o
Quoted 5 lines · Jeremy Allison

Turns out that ext4 doesn't suffer from the slowdown in the first place. The paper is extremly interesting, I'm looking at the implications for our default settings (most users are still using Samba on ext3 on Linux).

Quoted 5 lines

I thought the paper only talked about ext3, and theorized that delayed allocation in ext4 might be enough to make the problem go away, but they had not actually done any measurements to confirm this supposition. Has there been any more recent benchmarks comparing ext3, ext4, and XFS running Samba serving Windows clients?

It wouldn't be a bad idea to use this hint in the kernel to call fallocate(), given the fact that this is used by a number of apps (i.e. all of them) that predate fallocate().

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

#5 lore raw
Quoted 14 lines · Andreas Dilger
Quoted 7 lines · Theodore Ts'o
Quoted 5 lines · Jeremy Allison

Turns out that ext4 doesn't suffer from the slowdown in the first place. The paper is extremly interesting, I'm looking at the implications for our default settings (most users are still using Samba on ext3 on Linux).

Quoted 5 lines

I thought the paper only talked about ext3, and theorized that delayed allocation in ext4 might be enough to make the problem go away, but they had not actually done any measurements to confirm this supposition. Has there been any more recent benchmarks comparing ext3, ext4, and XFS running Samba serving Windows clients?

Quoted 3 lines

It wouldn't be a bad idea to use this hint in the kernel to call fallocate(), given the fact that this is used by a number of apps (i.e. all of them) that predate fallocate().

What, a one byte write that extends a file should be translated into an fallocate()? How.... crude. The question is, do we really want to be encouraging Microsoft in that way? :-)

Also, as it turns out, Microsoft is only doing this every 128k (i.e., touch one byte 128k after the end of the file, then write 128k of data, then write another 1 byte of garbage 128k past the end of the file, etc.), so ext4's delayed allocation algorithms seems to be able to handle things just fine.

I also suspect that if someone tried recompiling a kernel changing the value of EXT3_DEFAULT_RESERVE_BLOCKS from 8 to 32, or changing Samba to use the EXT3_IOC_SETRSVSZ ioctl immediately after opening a file for writing to set the block allocation reservation size for that inode to 32 blocks (128k), this might also enough of a kludge to solve most of the performance problems of Samba running on ext3 versus a Windows XP client. If someone *does* manage to try this experiment, please us know if it works...

- Ted

#6 lore raw
Quoted 38 lines · Theodore Tso
Quoted 14 lines · Andreas Dilger
Quoted 7 lines · Theodore Ts'o
Quoted 5 lines · Jeremy Allison

Turns out that ext4 doesn't suffer from the slowdown in the first place. The paper is extremly interesting, I'm looking at the implications for our default settings (most users are still using Samba on ext3 on Linux).

Quoted 5 lines

I thought the paper only talked about ext3, and theorized that delayed allocation in ext4 might be enough to make the problem go away, but they had not actually done any measurements to confirm this supposition. Has there been any more recent benchmarks comparing ext3, ext4, and XFS running Samba serving Windows clients?

Quoted 3 lines

It wouldn't be a bad idea to use this hint in the kernel to call fallocate(), given the fact that this is used by a number of apps (i.e. all of them) that predate fallocate().

What, a one byte write that extends a file should be translated into an fallocate()? How.... crude. The question is, do we really want to be encouraging Microsoft in that way? :-)

Also, as it turns out, Microsoft is only doing this every 128k (i.e., touch one byte 128k after the end of the file, then write 128k of data, then write another 1 byte of garbage 128k past the end of the file, etc.), so ext4's delayed allocation algorithms seems to be able to handle things just fine.

I also suspect that if someone tried recompiling a kernel changing the value of EXT3_DEFAULT_RESERVE_BLOCKS from 8 to 32, or changing Samba to use the EXT3_IOC_SETRSVSZ ioctl immediately after opening a file for writing to set the block allocation reservation size for that inode to 32 blocks (128k), this might also enough of a kludge to solve most of the performance problems of Samba running on ext3 versus a Windows XP client. If someone *does* manage to try this experiment, please us know if it works...

Quoted 2 lines

- Ted

Its not as simple as "the redirector does it every 128k". the
redirector does this but it varies from run to run and from client to client.
It is very common to see this happening in 60-64kb strides and other
strides as well.

It is probably some interaction with how large the actual i/o that the application did internally to the cache and some other thing. but anyway, it varies a lot. it is not always 128k.

#7 lore raw
Quoted 10 lines · ronnie sahlberg

Its not as simple as "the redirector does it every 128k". the
redirector does this but it varies from run to run and from client to client.
It is very common to see this happening in 60-64kb strides and other
strides as well.

It is probably some interaction with how large the actual i/o that the application did internally to the cache and some other thing. but anyway, it varies a lot. it is not always 128k.

Is there a maximum stride size used by the redirector? i.e., will it use something bigger than 128k? In any case, increasing the ext3 reservation window size should still be helpful. It's OK if we increase it to 32 blocks (128k, on a 4k block filesystem), and the stride size is smaller than that. But if it is often bigger than 128k, then then it would probably be better if samba used the EXT3_IOC_SETRSVSZ ioctl to dynamically set the reservation size as appropriate.

- Ted

#8 lore raw
Quoted 12 lines · Theodore Tso
Quoted 10 lines · ronnie sahlberg

Its not as simple as "the redirector does it every 128k". the
redirector does this but it varies from run to run and from client to client.
It is very common to see this happening in 60-64kb strides and other
strides as well.

It is probably some interaction with how large the actual i/o that the application did internally to the cache and some other thing. but anyway, it varies a lot. it is not always 128k.

Quoted 8 lines

Is there a maximum stride size used by the redirector? i.e., will it use something bigger than 128k? In any case, increasing the ext3 reservation window size should still be helpful. It's OK if we increase it to 32 blocks (128k, on a 4k block filesystem), and the stride size is smaller than that. But if it is often bigger than 128k, then then it would probably be better if samba used the EXT3_IOC_SETRSVSZ ioctl to dynamically set the reservation size as appropriate.

One might try to use "dd" from Cygwin on Windows. When I once analyzed this behaviour, the 1-byte writes were exactly at the end of the block the Win32 app gave to the kernel as seen by the sysinternals filemon tool.

Volker

#9 lore raw
Quoted 9 lines · Volker Lendecke
Quoted 8 lines

Is there a maximum stride size used by the redirector? i.e., will it use something bigger than 128k? In any case, increasing the ext3 reservation window size should still be helpful. It's OK if we increase it to 32 blocks (128k, on a 4k block filesystem), and the stride size is smaller than that. But if it is often bigger than 128k, then then it would probably be better if samba used the EXT3_IOC_SETRSVSZ ioctl to dynamically set the reservation size as appropriate.

Quoted 4 lines

One might try to use "dd" from Cygwin on Windows. When I once analyzed this behaviour, the 1-byte writes were exactly at the end of the block the Win32 app gave to the kernel as seen by the sysinternals filemon tool.

Hmmm, I always thought the network behavior was to prevent the app from being given an ENOSPACE error when it closed the file and the OS tried to flush the cache only to find that there was no space remaining on the remote file system to actually perform the writes the local cache accepted.

However, this suggests that the behavior is the same for local and remote file systems.

I do believe, as Ronnie said and as your evidence suggests, that these writes are at the end of the IO written into the cache by the app.

Regards
-------
Richard Sharpe, rsharpe[at]richardsharpe.com, rsharpe[at]samba.org, sharpe[at]ethereal.com, http://www.richardsharpe.com