patch and documentation
Hi,
The patch i attached defines max_sectors for each NBD device to be 256 which is 128KB as described in the protocol description i attached. The value 255 which is default in 2.4 kernels is not optimal. See my other posting in the lkml to read why.
The patch seems to work fine. My server-implementation receives one 128KB request as expected instead of two 127KB and 1Kb requests.
Even if 256 is the new max_sectors-default of kernel 2.6, the patch should be applied since the value should be part of the protocol specification and therefor part of nbd.c
The documentation i attached should be published somewhere. For example on nbd.sf.net since we didn't find one source where all the information is collected.
Thx
Sven
The NBD-protocol
How it works:
To use the NBD features of the kernel you must compile the kernel module
called "Network block device support" (CONFIG_BLK_DEV_NBD). After you've loaded the module the devices will appear in /dev/nbd/ if you're using devfs. If you're not using devfs you will possibly need to create your device-nodes with "mknod". All NBD devices have major ID 43. /dev/nbd/0 has minor ID 0, /dev/nbd/1 has minor ID 1 and so on.
To connect a NBD device to a remote server you need install the NBD tools downloadable at http://nbd.sf.net or http://sf.net/projects/nbd. Run "nbd-client <host> <tcp-port> /dev/nbd/0" to connect the device /dev/nbd/0 to the remote server. nbd-client will open a TCP-connection to the server and waits for some initial data which contains the size of the device. Than the handle of the TCP-connection is transferred to the kernel for further use. nbd-client will fork into background because the handle of the TCP-connection would be closed by the kernel if nbd-client exits. The tools also contain a very basic NBD server which will enable you to use any file or device as a NBD.
The protocol:
The protocol is based on top of TCP/IP. Both client and server send packets to each other. The server must send an init-packet to the client if a client connects. After that the server just receives request-packets from the client and sends back reply-packets until the connection is closed or a disconnect-request is received.
The amount of data that can be read or written with one request is limited to 128KB.
The current implementation of the NBD-protocol in the Linux-Kernel does send multiple requests without waiting for replies. So it does make sense for the server to handle requests in parallel.
Constants:
INIT_PASSWD = "NBDMAGIC"
INIT_MAGIC = 0x0000420281861253
REQUEST_MAGIC = 0x25609513
REPLY_MAGIC = 0x67446698
REQUEST_READ = 0
REQUEST_WRITE = 1
REQUEST_DISCONNECT = 2
Packets:
init-packet:
passwd : 8 bytes (string) = INIT_PASSWD
magic : 8 bytes (integer) = INIT_MAGIC
size : 8 bytes (integer) = size of the device in bytes
reserved : 128 bytes (filled with zeros)
request-packet:
magic : 4 bytes (integer) = REQUEST_MAGIC
type : 4 bytes (integer) = REQUEST_READ, REQUEST_WRITE or REQUEST_DISCONNECT
handle : 8 bytes (integer) = request-handle
from : 8 bytes (integer) = start offset for read/write-operation in bytes
length : 4 bytes (integer) = length of the read/write-operationion bytes
data : x bytes (only for write request, x = length field of this packet)
reply-packet:
magic : 4 bytes (integer) = REPLY_MAGIC
error : 4 bytes (integer) = errorcode (0 = no error) handle : 8 bytes (integer) = copy of request-handle
data : x bytes (only for reply to read request and if no error occured,
x = length field of the request packet)
all integer values are stored unsigned and in network-byte-order (big-endian)
Hi!
Paul, do you think this docs could be added to the end of Documentation/nbd.txt?
The patch also looks harmless enough for applying ;-).
Pavel
Quoted 87 lines
The patch i attached defines
max_sectorsfor each NBD device to be 256 which is 128KB as described in the protocol description i attached. The value 255 which is default in 2.4 kernels is not optimal. See my other posting in the lkml to read why.
The patch seems to work fine. My server-implementation receives one 128KB request as expected instead of two 127KB and 1Kb requests.Even if 256 is the new max_sectors-default of kernel 2.6, the patch should be applied since the value should be part of the protocol specification and therefor part of
nbd.cThe documentation i attached should be published somewhere. For example on nbd.sf.net since we didn't find one source where all the information is collected.
Thx
SvenThe NBD-protocol
How it works:
To use the NBD features of the kernel you must compile the kernel module
called "Network block device support" (CONFIG_BLK_DEV_NBD). After you've loaded the module the devices will appear in/dev/nbd/if you're using devfs. If you're not using devfs you will possibly need to create your device-nodes with "mknod". All NBD devices have major ID 43./dev/nbd/0has minor ID 0,/dev/nbd/1has minor ID 1 and so on.
To connect a NBD device to a remote server you need install the NBD tools downloadable at http://nbd.sf.net or http://sf.net/projects/nbd. Run "nbd-client <host> <tcp-port>/dev/nbd/0" to connect the device/dev/nbd/0to the remote server. nbd-client will open a TCP-connection to the server and waits for some initial data which contains the size of the device. Than the handle of the TCP-connection is transferred to the kernel for further use. nbd-client will fork into background because the handle of the TCP-connection would be closed by the kernel if nbd-client exits. The tools also contain a very basic NBD server which will enable you to use any file or device as a NBD.The protocol:
The protocol is based on top of TCP/IP. Both client and server send packets to each other. The server must send an init-packet to the client if a client connects. After that the server just receives request-packets from the client and sends back reply-packets until the connection is closed or a disconnect-request is received.The amount of data that can be read or written with one request is limited to 128KB.
The current implementation of the NBD-protocol in the Linux-Kernel does send multiple requests without waiting for replies. So it does make sense for the server to handle requests in parallel.
Constants:
INIT_PASSWD= "NBDMAGIC"
INIT_MAGIC=0x0000420281861253
REQUEST_MAGIC=0x25609513
REPLY_MAGIC=0x67446698
REQUEST_READ= 0
REQUEST_WRITE= 1
REQUEST_DISCONNECT= 2Packets:
init-packet:
passwd : 8 bytes (string) =INIT_PASSWD
magic : 8 bytes (integer) =INIT_MAGIC
size : 8 bytes (integer) = size of the device in bytes
reserved : 128 bytes (filled with zeros)request-packet:
magic : 4 bytes (integer) =REQUEST_MAGIC
type : 4 bytes (integer) =REQUEST_READ,REQUEST_WRITEorREQUEST_DISCONNECT
handle : 8 bytes (integer) = request-handle
from : 8 bytes (integer) = start offset for read/write-operation in bytes
length : 4 bytes (integer) = length of the read/write-operationion bytes
data : x bytes (only for write request, x = length field of this packet)reply-packet:
magic : 4 bytes (integer) =REPLY_MAGIC
error : 4 bytes (integer) = errorcode (0 = no error) handle : 8 bytes (integer) = copy of request-handle
data : x bytes (only for reply to read request and if no error occured,
x = length field of the request packet)all integer values are stored unsigned and in network-byte-order (big-endian)
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
Quoted 2 lines
Paul, do you think this docs could be added to the end of
Documentation/nbd.txt?
Well, i noticed that i used tabs in some places. sorry for that. the tabs should be replaced by 4 spaces.
Quoted 1 line
The patch also looks harmless enough for applying ;-).
I take it as a compliment.
Quoted 2 lines · Pavel Machek
Paul, do you think this docs could be added to the end of
Documentation/nbd.txt?
I've made some changes to 2.6 nbd.txt, so let me look at integrating this into those changes and also putting them into 2.4.
Quoted 1 line
The patch also looks harmless enough for applying ;-).
Harmless enough, although I'm not sure it really makes that much difference. The max_sectors being set to 255 doesn't, by itself, explain the back and forth 127k, 1k request thing. Typically what you'll see is 127k, 127k, 127k, etc. and then some odd sized request at the end. Or the device gets unplugged anyway at some point and there are odd sized requests scattered throughout...that's especially going to be true if the reads or writes are from an actual disk, rather than /dev/null. I may be just coincidence that setting max_sectors to 256 actually helps. Also, are we sure that all those requests you're seeing are of the same type (all reads, all writes)?
--
Paul
Quoted 95 lines
Quoted 78 lines
The patch i attached defines
max_sectorsfor each NBD device to be 256 which is 128KB as described in the protocol description i attached. The value 255 which is default in 2.4 kernels is not optimal. See my other posting in the lkml to read why.
The patch seems to work fine. My server-implementation receives one 128KB request as expected instead of two 127KB and 1Kb requests.Even if 256 is the new max_sectors-default of kernel 2.6, the patch should be applied since the value should be part of the protocol specification and therefor part of
nbd.cThe documentation i attached should be published somewhere. For example on nbd.sf.net since we didn't find one source where all the information is collected.
Thx
SvenThe NBD-protocol
How it works:
To use the NBD features of the kernel you must compile the kernel module
called "Network block device support" (CONFIG_BLK_DEV_NBD). After you've loaded the module the devices will appear in/dev/nbd/if you're using devfs. If you're not using devfs you will possibly need to create your device-nodes with "mknod". All NBD devices have major ID 43./dev/nbd/0has minor ID 0,/dev/nbd/1has minor ID 1 and so on.
To connect a NBD device to a remote server you need install the NBD tools downloadable at http://nbd.sf.net or http://sf.net/projects/nbd. Run "nbd-client <host> <tcp-port>/dev/nbd/0" to connect the device/dev/nbd/0to the remote server. nbd-client will open a TCP-connection to the server and waits for some initial data which contains the size of the device. Than the handle of the TCP-connection is transferred to the kernel for further use. nbd-client will fork into background because the handle of the TCP-connection would be closed by the kernel if nbd-client exits. The tools also contain a very basic NBD server which will enable you to use any file or device as a NBD.The protocol:
The protocol is based on top of TCP/IP. Both client and server send packets to each other. The server must send an init-packet to the client if a client connects. After that the server just receives request-packets from the client and sends back reply-packets until the connection is closed or a disconnect-request is received.The amount of data that can be read or written with one request is limited to 128KB.
The current implementation of the NBD-protocol in the Linux-Kernel does send multiple requests without waiting for replies. So it does make sense for the server to handle requests in parallel.
Constants:
INIT_PASSWD= "NBDMAGIC"
INIT_MAGIC=0x0000420281861253
REQUEST_MAGIC=0x25609513
REPLY_MAGIC=0x67446698
REQUEST_READ= 0
REQUEST_WRITE= 1
REQUEST_DISCONNECT= 2Packets:
init-packet:
passwd : 8 bytes (string) =INIT_PASSWD
magic : 8 bytes (integer) =INIT_MAGIC
size : 8 bytes (integer) = size of the device in bytes
reserved : 128 bytes (filled with zeros)request-packet:
magic : 4 bytes (integer) =REQUEST_MAGIC
type : 4 bytes (integer) =REQUEST_READ,REQUEST_WRITEorREQUEST_DISCONNECT
handle : 8 bytes (integer) = request-handle
from : 8 bytes (integer) = start offset for read/write-operation in bytes
length : 4 bytes (integer) = length of the read/write-operationion bytes
data : x bytes (only for write request, x = length field of this packet)Quoted 8 lines
reply-packet:
magic : 4 bytes (integer) =REPLY_MAGIC
error : 4 bytes (integer) = errorcode (0 = no error) handle : 8 bytes (integer) = copy of request-handle
data : x bytes (only for reply to read request and if no error occured,
x = length field of the request packet)all integer values are stored unsigned and in network-byte-order (big-endian)
--
When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 12 lines
Quoted 1 line
The patch also looks harmless enough for applying ;-).
Harmless enough, although I'm not sure it really makes that much difference. The
max_sectorsbeing set to 255 doesn't, by itself, explain the back and forth 127k, 1k request thing. Typically what you'll see is 127k, 127k, 127k, etc. and then some odd sized request at the end. Or the device gets unplugged anyway at some point and there are odd sized requests scattered throughout...that's especially going to be true if the reads or writes are from an actual disk, rather than/dev/null. I may be just coincidence that settingmax_sectorsto 256 actually helps. Also, are we sure that all those requests you're seeing are of the same type (all reads, all writes)?
Well, i guess the cache uses a value of 256 sectors to do read-ahead and such. I used dd if=/dev/nbd/0 of=/dev/null bs=X with both X=1 and X=1M. Both with the same result. That the 1byte requests join together to bigger ones can only be explained with read-aheads strategies. Anyway, the result is always the same:
without patch: 127KB, 1KB, 127KB, 1KB
with path: 128KB, 128KB, 128KB
As long as dd doesn't write i'm sure that i didn't see any write requests. In addition it is a very regular pattern.
If it is really the case that the cache reads 256 sectors and the default limit is 255, than this would also happen for all other block-devices. In addition it would be a good thing to look up if the cache takes the max_sectors stuff into accout while determining the amout of sectors it reads ahead.
Quoted 2 lines · Sven Köhler
Well, i guess the cache uses a value of 256 sectors to do read-ahead and such.
Well it sounds like the real problem here is the vm_max_readahead setting then. Try this:
cat /proc/sys/vm/max-readahead
Probably, it's set to 31 on your system.
Try something like the following:
echo "126" > /proc/sys/vm/max-readahead
I think that should help out.
--
Paul
Quoted 5 lines
Quoted 2 lines
Well, i guess the cache uses a value of 256 sectors to do read-ahead and such.
Well it sounds like the real problem here is the
vm_max_readaheadsetting then. Try this:
I will try it, although i think that i'm using the deafult values.
Anyway: the NBD module should set the max_sectors to a certain value - i chose 256 sectors. Perhaps Pavel or Paul may decide to use a higher ot smaller value. A limit should be part of the protocol or handshaked when connecting to the server (what is not possible without changing the protocol)
Hi!
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
Well, i guess the cache uses a value of 256 sectors to do read-ahead and such.
Well it sounds like the real problem here is the
vm_max_readaheadsetting then. Try this:
Quoted 7 lines
I will try it, although i think that i'm using the deafult values.
Anyway: the NBD module should set the
max_sectorsto a certain value - i chose 256 sectors. Perhaps Pavel or Paul may decide to use a higher ot smaller value. A limit should be part of the protocol or handshaked when connecting to the server (what is not possible without changing the protocol)
I do not see a reason it should be handshaked. IMNSHO we should simply say that no request should be bigger than 1MB in protocol, make sure that kernel does <=128KB requests, and make sure nbd-servers can handle 1MB, and be done with that.
Pavel
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
Another idea would be to be abled to specify the max_sectors while connecting an NBD. That would add an optional paramter to the nbd-client command line. (like it is possible for the blocksize)
Hi!
Quoted 3 lines
Another idea would be to be abled to specify the
max_sectorswhile connecting an NBD. That would add an optional paramter to the nbd-client command line. (like it is possible for the blocksize)
I do not see why it should be configurable...
Pavel
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
Quoted 3 lines
Quoted 1 line
Another idea would be to be abled to specify the
max_sectorswhile connecting an NBD. That would add an optional paramter to the nbd-client command line. (like it is possible for the blocksize)I do not see why it should be configurable...
We may regret to use a certain value, although i agree that 1MB should be sufficient for the future.
Hi!
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Another idea would be to be abled to specify the
max_sectorswhile connecting an NBD. That would add an optional paramter to the nbd-client command line. (like it is possible for the blocksize)I do not see why it should be configurable...
We may regret to use a certain value, although i agree that 1MB should be sufficient for the future.
I believe that 1MB is good, and good enough for close future. If that ever proves to be problem, we can add handshake at that point. But I do not believe it will be neccessary.
Pavel
Signature · 2 lines
When do you have a heart between your knees? [Johanka's followup: and *two* hearts?]
Quoted 3 lines
I believe that 1MB is good, and good enough for close future. If that ever proves to be problem, we can add handshake at that point. But I do not believe it will be neccessary.
Well, we shouldn't discuss the max_sectors problem (or what ever we should call it) too much. I must seem as a bullhead to you.
Quoted 7 lines · Pavel Machek
Hi!
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Another idea would be to be abled to specify the
max_sectorswhile connecting an NBD. That would add an optional paramter to the nbd-client command line. (like it is possible for the blocksize)
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
I do not see why it should be configurable...
We may regret to use a certain value, although i agree that 1MB should be sufficient for the future.
I believe that 1MB is good, and good enough for close future. If that ever proves to be problem, we can add handshake at that point. But I do not believe it will be neccessary.
But, who ever said the buffer in the nbd-server had to be statically allocated? I have a version of nbd-server that is modified to handle any size request that the client side throws at it -- if the buffer is not large enough, it simply reallocates it.
--
Paul
Quoted 8 lines
Quoted 3 lines
I believe that 1MB is good, and good enough for close future. If that ever proves to be problem, we can add handshake at that point. But I do not believe it will be neccessary.
But, who ever said the buffer in the nbd-server had to be statically allocated? I have a version of nbd-server that is modified to handle any size request that the client side throws at it -- if the buffer is not large enough, it simply reallocates it.
Well, imagine that somebody develops a server implementation (well, that was what some friends and me did in the past few days) than it is just good to know, that there is a limit for the length field of the request-packet. If there is no limit, the server implementation has to be abled to answer requests of any size. That is very complicated. For example if the server has only 64MB of memeory, and the kernel would be abled to send a 128MB request, how would you handle that?
Quoted 2 lines
echo "126" >
/proc/sys/vm/max-readahead
I think that should help out.
Well, it didn't help. Still the same funny pattern of 127 and 1KB requests ;-)