2.5.8 IDE 36]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · Alan Cox
Quoted 1 line
We need to support partitioning on loopback devices in that case.
No, you just need to do the loopback over nbd - you need something to do the byte swapping anyway (ie you can't really use the normal "loop"
device: I really just meant the more generic "loop the data back" approach).nbd devices already do partitioning, I'm fairly certain.
Well, sort of. They make the right motions in the code, waving their arms about in the right directions. It's doubtful if what's there is enough.
Quoted 1 line
(But no, I've never tested it, obviously).
Neither have the authors.
Quoted 4 lines
Btw, while I'm at it - who out there actually uses the new "enbd" (Enhanced NBD)? I have this feeling that that would be the better choice, since unlike plain nbd it should be deadlock-free on localhost (ie you don't need a remote machine).
No - it will deadlock the same way. The deadlock is not related to the device driver itself. It goes like this.
kernel runs low on memory
kernel flushes buffers to device drivers under pressure
nbd client daemon sends to the net (potential conflict with tcp buffers,
but not the mechanism that hurts)
server on the _same machine_ receives request over the net server tries to write request to disk
server process needs buffers to write to
bang (or whatever deadlock sounds like)
The enbd client has a special "-a" flag which is meant to be used in cases where we trust the transport medium. Localhost is such a case.
With it, the protocol in the client daemon will ack the kernel _before_ it gets an ack from the remote server. That should help relieve the deadlock. Things then go like this:
kernel runs low on memory
kernel flushes buffers to device drivers under pressure nbd client daemon sends to the net
* client acks kernel and releases buffers in kernel
server on the _same machine_ receives request over the net server tries to write request to disk
server process needs buffers to write to
* server gets buffers released by client in kernel
At least, potentially. If I recall right, there's still a deadlock window in-kernel, but it's small. I don't recall the details. Oh - well, the request still hangs around in the driver until the client daemon acks .. maybe the client daemon can't ack without being swapped in first, and that'd be deadlock.
Well, there's a "-s" flag (for swap devices) that does an mlockall() that might take care of that. So "-a -s" might do it. Really needs a "-aa" option (release write request in kernel asap).
However, a real cure would be not to use the network transport at all in enbd. Enbd can use different transports. Instead of using the stub for the server in the client daemon code, one could use the "fileserver" object directly. It'd just be a question of changing the initialization. Both have the same generic interface.
Incidentally, the newer VM code in 2.4 is a real problem for enbd. I would ideally like _no_ _buffering_ whatsoever on the clientside.
As it is, it seems almost mandatory to raise the bdflush sync boundary as high as possible, and to drop the async boundary really low. One wants to start sending buffers out across the net asynchronously as early as possible, and one never wants to go sync. Going sync makes for a deadlock where we need to send a request across the net in order to free a buffer, but we can't, because we need some buffers to do that with ...
Is there any way of turning off the VM for a single device.
The 2.5 i/o changes, otoh, seem immensely favourable to enbd, since it's multithreaded and asyncronous in-kernel.
Peter
Hi!
Quoted 19 lines
With it, the protocol in the client daemon will ack the kernel _before_ it gets an ack from the remote server. That should help relieve the deadlock. Things then go like this:
kernel runs low on memory
kernel flushes buffers to device drivers under pressure nbd client daemon sends to the net
* client acks kernel and releases buffers in kernel
server on the_samemachine_receives request over the net server tries to write request to disk
server process needs buffers to write to
* server gets buffers released by client in kernelAt least, potentially. If I recall right, there's still a deadlock window in-kernel, but it's small. I don't recall the details. Oh - well, the request still hangs around in the driver until the client daemon acks .. maybe the client daemon can't ack without being swapped in first, and that'd be deadlock.
Quoted 3 lines
Well, there's a "-s" flag (for swap devices) that does an
mlockall()that might take care of that. So "-a -s" might do it. Really needs a "-aa" option (release write request in kernel asap).
Well, with mlockall(), I'd believe it could be made to work. Okay.
Pavel
Signature · 2 lines
(about SSSCA) "I don't say this lightly. However, I really think that the U.S. no longer is classifiable as a democracy, but rather as a plutocracy." --hpa