Samba speed
Here's a really interesting paper from Intel that they recently brought to my attention.
http://software.intel.com/en-us/articles/windows-client-cifs-behavior-can-slow-linux-nas-performance
Looks like using XFS for your Linux Samba server, or setting "strict allocate = yes" can make a big difference due to sparse file issues.
Comments welcome !
Jeremy.
Quoted 8 lines · Jeremy Allison
Here's a really interesting paper from Intel that they recently brought to my attention.
http://software.intel.com/en-us/articles/windows-client-cifs-behavior-can-slow-linux-nas-performance
Looks like using XFS for your Linux Samba server, or setting "strict allocate = yes" can make a big difference due to sparse file issues.
Glibc 2.7 (as shipped in Ubuntu Hardy) has posix_fallocate wired up to the fallocate system call, and ext4 supports delayed allocation as well as preallocation. There are number of userspace applications --- rsync, samba, and most bittorrent applications come to mind --- where use of fallocate would be a big win.
- Ted
Quoted 10 lines · Theodore Tso
Quoted 8 lines · Jeremy Allison
Here's a really interesting paper from Intel that they recently brought to my attention.
http://software.intel.com/en-us/articles/windows-client-cifs-behavior-can-slow-linux-nas-performance
Looks like using XFS for your Linux Samba server, or setting "strict allocate = yes" can make a big difference due to sparse file issues.
Quoted 5 lines
Glibc 2.7 (as shipped in Ubuntu Hardy) has
posix_fallocatewired up to the fallocate system call, and ext4 supports delayed allocation as well as preallocation. There are number of userspace applications --- rsync, samba, and most bittorrent applications come to mind --- where use of fallocate would be a big win.
Turns out that ext4 doesn't suffer from the slowdown in the first place. The paper is extremly interesting, I'm looking at the implications for our default settings (most users are still using Samba on ext3 on Linux).
Jeremy.
Quoted 5 lines · Jeremy Allison
Turns out that ext4 doesn't suffer from the slowdown in the first place. The paper is extremly interesting, I'm looking at the implications for our default settings (most users are still using Samba on ext3 on Linux).
I thought the paper only talked about ext3, and theorized that delayed allocation in ext4 might be enough to make the problem go away, but they had not actually done any measurements to confirm this supposition. Has there been any more recent benchmarks comparing ext3, ext4, and XFS running Samba serving Windows clients?
- Ted
P.S. I'll be on the Google campus tomorrow and Wednesday attending the Ubuntu developer's conference; we should get together for lunch or dinner or some such....
Quoted 7 lines · Theodore Ts'o
Quoted 5 lines · Jeremy Allison
Turns out that ext4 doesn't suffer from the slowdown in the first place. The paper is extremly interesting, I'm looking at the implications for our default settings (most users are still using Samba on ext3 on Linux).
Quoted 5 lines
I thought the paper only talked about ext3, and theorized that delayed allocation in ext4 might be enough to make the problem go away, but they had not actually done any measurements to confirm this supposition. Has there been any more recent benchmarks comparing ext3, ext4, and XFS running Samba serving Windows clients?
It wouldn't be a bad idea to use this hint in the kernel to call fallocate(), given the fact that this is used by a number of apps (i.e. all of them) that predate fallocate().
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Quoted 14 lines · Andreas Dilger
Quoted 7 lines · Theodore Ts'o
Quoted 5 lines · Jeremy Allison
Turns out that ext4 doesn't suffer from the slowdown in the first place. The paper is extremly interesting, I'm looking at the implications for our default settings (most users are still using Samba on ext3 on Linux).
Quoted 5 lines
I thought the paper only talked about ext3, and theorized that delayed allocation in ext4 might be enough to make the problem go away, but they had not actually done any measurements to confirm this supposition. Has there been any more recent benchmarks comparing ext3, ext4, and XFS running Samba serving Windows clients?
Quoted 3 lines
It wouldn't be a bad idea to use this hint in the kernel to call
fallocate(), given the fact that this is used by a number of apps (i.e. all of them) that predatefallocate().
What, a one byte write that extends a file should be translated into an fallocate()? How.... crude. The question is, do we really want to be encouraging Microsoft in that way? :-)
Also, as it turns out, Microsoft is only doing this every 128k (i.e., touch one byte 128k after the end of the file, then write 128k of data, then write another 1 byte of garbage 128k past the end of the file, etc.), so ext4's delayed allocation algorithms seems to be able to handle things just fine.
I also suspect that if someone tried recompiling a kernel changing the value of EXT3_DEFAULT_RESERVE_BLOCKS from 8 to 32, or changing Samba to use the EXT3_IOC_SETRSVSZ ioctl immediately after opening a file for writing to set the block allocation reservation size for that inode to 32 blocks (128k), this might also enough of a kludge to solve most of the performance problems of Samba running on ext3 versus a Windows XP client. If someone *does* manage to try this experiment, please us know if it works...
- Ted
Quoted 38 lines · Theodore Tso
Quoted 14 lines · Andreas Dilger
Quoted 7 lines · Theodore Ts'o
Quoted 5 lines · Jeremy Allison
Turns out that ext4 doesn't suffer from the slowdown in the first place. The paper is extremly interesting, I'm looking at the implications for our default settings (most users are still using Samba on ext3 on Linux).
Quoted 5 lines
I thought the paper only talked about ext3, and theorized that delayed allocation in ext4 might be enough to make the problem go away, but they had not actually done any measurements to confirm this supposition. Has there been any more recent benchmarks comparing ext3, ext4, and XFS running Samba serving Windows clients?
Quoted 3 lines
It wouldn't be a bad idea to use this hint in the kernel to call
fallocate(), given the fact that this is used by a number of apps (i.e. all of them) that predatefallocate().What, a one byte write that extends a file should be translated into an
fallocate()? How.... crude. The question is, do we really want to be encouraging Microsoft in that way? :-)Also, as it turns out, Microsoft is only doing this every 128k (i.e., touch one byte 128k after the end of the file, then write 128k of data, then write another 1 byte of garbage 128k past the end of the file, etc.), so ext4's delayed allocation algorithms seems to be able to handle things just fine.
I also suspect that if someone tried recompiling a kernel changing the value of
EXT3_DEFAULT_RESERVE_BLOCKSfrom 8 to 32, or changing Samba to use theEXT3_IOC_SETRSVSZioctl immediately after opening a file for writing to set the block allocation reservation size for that inode to 32 blocks (128k), this might also enough of a kludge to solve most of the performance problems of Samba running on ext3 versus a Windows XP client. If someone *does* manage to try this experiment, please us know if it works...
Quoted 2 lines
- Ted
Its not as simple as "the redirector does it every 128k". the
redirector does this but it varies from run to run and from client to client.
It is very common to see this happening in 60-64kb strides and other
strides as well.
It is probably some interaction with how large the actual i/o that the application did internally to the cache and some other thing. but anyway, it varies a lot. it is not always 128k.
Quoted 10 lines · ronnie sahlberg
Its not as simple as "the redirector does it every 128k". the
redirector does this but it varies from run to run and from client to client.
It is very common to see this happening in 60-64kb strides and other
strides as well.It is probably some interaction with how large the actual i/o that the application did internally to the cache and some other thing. but anyway, it varies a lot. it is not always 128k.
Is there a maximum stride size used by the redirector? i.e., will it use something bigger than 128k? In any case, increasing the ext3 reservation window size should still be helpful. It's OK if we increase it to 32 blocks (128k, on a 4k block filesystem), and the stride size is smaller than that. But if it is often bigger than 128k, then then it would probably be better if samba used the EXT3_IOC_SETRSVSZ ioctl to dynamically set the reservation size as appropriate.
- Ted
Quoted 12 lines · Theodore Tso
Quoted 10 lines · ronnie sahlberg
Its not as simple as "the redirector does it every 128k". the
redirector does this but it varies from run to run and from client to client.
It is very common to see this happening in 60-64kb strides and other
strides as well.It is probably some interaction with how large the actual i/o that the application did internally to the cache and some other thing. but anyway, it varies a lot. it is not always 128k.
Quoted 8 lines
Is there a maximum stride size used by the redirector? i.e., will it use something bigger than 128k? In any case, increasing the ext3 reservation window size should still be helpful. It's OK if we increase it to 32 blocks (128k, on a 4k block filesystem), and the stride size is smaller than that. But if it is often bigger than 128k, then then it would probably be better if samba used the
EXT3_IOC_SETRSVSZioctl to dynamically set the reservation size as appropriate.
One might try to use "dd" from Cygwin on Windows. When I once analyzed this behaviour, the 1-byte writes were exactly at the end of the block the Win32 app gave to the kernel as seen by the sysinternals filemon tool.
Volker
Quoted 9 lines · Volker Lendecke
Quoted 8 lines
Is there a maximum stride size used by the redirector? i.e., will it use something bigger than 128k? In any case, increasing the ext3 reservation window size should still be helpful. It's OK if we increase it to 32 blocks (128k, on a 4k block filesystem), and the stride size is smaller than that. But if it is often bigger than 128k, then then it would probably be better if samba used the
EXT3_IOC_SETRSVSZioctl to dynamically set the reservation size as appropriate.
Quoted 4 lines
One might try to use "dd" from Cygwin on Windows. When I once analyzed this behaviour, the 1-byte writes were exactly at the end of the block the Win32 app gave to the kernel as seen by the sysinternals filemon tool.
Hmmm, I always thought the network behavior was to prevent the app from being given an ENOSPACE error when it closed the file and the OS tried to flush the cache only to find that there was no space remaining on the remote file system to actually perform the writes the local cache accepted.
However, this suggests that the behavior is the same for local and remote file systems.
I do believe, as Ronnie said and as your evidence suggests, that these writes are at the end of the IO written into the cache by the app.
Regards
-------
Richard Sharpe, rsharpe[at]richardsharpe.com, rsharpe[at]samba.org, sharpe[at]ethereal.com, http://www.richardsharpe.com