Skip to contentSkip to conversation

ENBD for 2.5.64

PATCH Peter T. Breuer 8 participants 16 messages Mar 25, 2003 – Mar 30, 2003 2 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 1048623613.25914.14.camel@lotte

#1 lore raw

"Justin Cormack wrote:"

Quoted 11 lines · Peter T. Breuer
Quoted 4 lines
Quoted 4 lines

ENBD is not a replacement for NBD - the two are alternatives, aimed at different niches. ENBD is a sort of heavyweight industrial NBD. It does many more things and has a different achitecture. Kernel NBD is like a stripped down version of ENBD. Both should be in the kernel.

hmm, I would argue that nbd is ok, as it is a nice lightweight block device (though I have not been able to use it due to the fact that I can never find a userspace and kernel that work together), while ENBD should be replaced by iscsi, now that is a real ietf standard, and can burn CDs across the net and all that extra stuff.

It's not a bad idea. But ENBD in particular can use any transport, precisely becuase its networking is done in userspace. One only has to write a stream.c module for it that implements

read
write
open
close

(There are currently implementations for three transports, including tcp of course).

Quoted 2 lines

And I am intending to write an iscsi client sometime, but it got delayed. The server stuff is already available from 3com.

Possibly, but ENBD is designed to fail :-). And networks fail. What will your iscsi implementation do when somebody resets the router? All those issues are handled by ENBD. ENBD breaks off and reconnects automatically. It reacts right to removable media.

I should also have said that ENBD has the following features (I said I forgot some!)

9) it drops into a mode where it md5sums both ends and skips writes of equal blocks, if that's faster. It moves in and out of this mode automatically. This helps RAID resyncs (2* overspeed is common on 100BT nets, that is 20MB/s.).

10) integration with RAID - it advises raid correctly of its state and does hot add and remove correctly (well, you need my patches to raid, but there you are ...).

Of course, if somebody wants me to make enbd appear like a scis device instead of a generic block device, I guess I could do that. Except, that yecch, I have seen the scsi code, and I do not understand it.

Another good idea is to make the wire protocol iscsi compatible. I have no objection to that.

Peter

#2 lore raw
Quoted 9 lines · Peter T. Breuer

"Justin Cormack wrote:"

Quoted 2 lines

And I am intending to write an iscsi client sometime, but it got delayed. The server stuff is already available from 3com.

Possibly, but ENBD is designed to fail :-). And networks fail. What will your iscsi implementation do when somebody resets the router? All those issues are handled by ENBD. ENBD breaks off and reconnects automatically. It reacts right to removable media.

Yeah, iSCSI handles all that and more. It's a behemoth of a specification. (whether a particular implementation implements all that stuff correctly is another matter...)

BTW, I'm a big enbd fan :) I like enbd for it's _simplicity_ compared to iSCSI.

Jeff

#3 lore raw
Quoted 6 lines · Jeff Garzik
Quoted 5 lines · Peter T. Breuer

"Justin Cormack wrote:"

Quoted 2 lines

And I am intending to write an iscsi client sometime, but it got delayed. The server stuff is already available from 3com.

Quoted 8 lines
Quoted 4 lines

Possibly, but ENBD is designed to fail :-). And networks fail. What will your iscsi implementation do when somebody resets the router? All those issues are handled by ENBD. ENBD breaks off and reconnects automatically. It reacts right to removable media.

Yeah, iSCSI handles all that and more. It's a behemoth of a specification. (whether a particular implementation implements all that stuff correctly is another matter...)

Indeed, there are iSCSI implementations that do multipath and failover.

Both iSCSI and ENBD currently have issues with pending writes during network outages. The current I/O layer fails to report failed writes to fsync and friends.

Quoted 2 lines

BTW, I'm a big enbd fan :) I like enbd for it's _simplicity_ compared to iSCSI.

Definitely. The iSCSI protocol is more powerful but _much_ more complex than ENBD. I've spent two years working on iSCSI but guess which I use at home..

Signature · 1 line
Matt Mackall : http://www.selenic.com : of or relating to the moon
#4 lore raw

"A month of sundays ago Matt Mackall wrote:"

Quoted 11 lines · Jeff Garzik
Quoted 10 lines
Quoted 8 lines · Peter T. Breuer

"Justin Cormack wrote:"

Quoted 2 lines

And I am intending to write an iscsi client sometime, but it got delayed. The server stuff is already available from 3com.

Possibly, but ENBD is designed to fail :-). And networks fail. What will your iscsi implementation do when somebody resets the router? All those issues are handled by ENBD. ENBD breaks off and reconnects automatically. It reacts right to removable media.

Quoted 6 lines
Quoted 3 lines

Yeah, iSCSI handles all that and more. It's a behemoth of a specification. (whether a particular implementation implements all that stuff correctly is another matter...)

Indeed, there are iSCSI implementations that do multipath and failover.

Somebody really ought to explain it to me :-). I can't keep up with all this!

Quoted 3 lines

Both iSCSI and ENBD currently have issues with pending writes during network outages. The current I/O layer fails to report failed writes to fsync and friends.

ENBD has two (configurable) behaviors here. Perhaps it should have more. By default it blocks pending reads and writes during times when the connection is down. It can be configured to error them instead. The erroring behavior is what you want when running under soft RAID, as it's raid that should do the deciding about how to treat the requests according to the overall state of the array, so it needs definite yes/no info on each request, no "maybe".

Perhaps in a third mode requests should be blocked and time out after about half an hour (or some number, in an infinite spectrum).

What I would like is some way of telling how backed up the VM is against us. If the VM is full of dirty buffers aimed at us, then I think we should consider erroring instead of blocking. The problem is that at that point we're likely not getting any requests at all, because the kernel long ago ran out of the 256 requests it has in hand to send us.

There is indeed an information disconnect with VMS in those circumstances that I've never known how to solve. FSs too are a problem, because unless they are mounted sync they will happily permit writes to a file on a fs on a blocked device even if that fills the machine with buffers that can't go anywhere. Among other things that will run tcp out of buffer space that is necessary in order to flush those buffers even if the connection does come back. And even if the mount is sync then some fs's (e.g. ext2) still allow infinitely much writing to a blocked device under some circumstances (start two processes writing to the same file .. the second will write to buffers).

Peter

#5 lore raw
Quoted 7 lines
Quoted 3 lines

Both iSCSI and ENBD currently have issues with pending writes during network outages. The current I/O layer fails to report failed writes to fsync and friends.

ENBD has two (configurable) behaviors here. Perhaps it should have more. By default it blocks pending reads and writes during times when the connection is down. It can be configured to error them instead.

And in this case, the upper layers will silently drop write errors on current kernels.

Cisco's Linux iSCSI driver has a configurable timeout, defaulting to 'infinite', btw.

Quoted 6 lines

What I would like is some way of telling how backed up the VM is against us. If the VM is full of dirty buffers aimed at us, then I think we should consider erroring instead of blocking. The problem is that at that point we're likely not getting any requests at all, because the kernel long ago ran out of the 256 requests it has in hand to send us.

Hrrmm. The potential to lose data by surprise here is not terribly appealing. It might be better to add an accounting mechanism to say "never go above x dirty pages against block device n" or something of the sort but you can still get into trouble if you happen to have hundreds of iSCSI devices each with their own request queue..

Signature · 1 line
Matt Mackall : http://www.selenic.com : of or relating to the moon
#6 lore raw

"A month of sundays ago Matt Mackall wrote:"

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Both iSCSI and ENBD currently have issues with pending writes during network outages. The current I/O layer fails to report failed writes to fsync and friends.

ENBD has two (configurable) behaviors here. Perhaps it should have more. By default it blocks pending reads and writes during times when the connection is down. It can be configured to error them instead.

Quoted 5 lines

And in this case, the upper layers will silently drop write errors on current kernels.

Cisco's Linux iSCSI driver has a configurable timeout, defaulting to 'infinite', btw.

That corresponds to enbd's default behavior. Sigh. Guess I'll have to make it 0-infty, instead of 0 or infty. It's easy enough - just need to make it settable in proc (and think about which is the one line I need to touch ...).

Quoted 2 lines

Hrrmm. The potential to lose data by surprise here is not terribly appealing.

Making the driver "intelligent" is indeed bad news for the more intelligent admin. I was thinking of making it default to 0 timeout if it knows it's running under raid, but I have a natural antipathy to such in-driver decisions. My conscience would be slightly less on alert if the userspace daemon did the decision-making. I suppose it could.

Quoted 4 lines

It might be better to add an accounting mechanism to say
"never go above x dirty pages against block device n" or something of the sort but you can still get into trouble if you happen to have hundreds of iSCSI devices each with their own request queue..

Well, you can get in trouble if you allow even a single dirty page to be outstanding to something, and have thousands of those somethings.

That's not the normal situation, however, whereas it is normal to have a single network device and to be writing pell-mell to it oblivious to the state of the device itself.

Peter

#7 lore raw
In reply to Matt Mackall
Quoted 11 lines · Matt Mackall
Quoted 3 lines

Yeah, iSCSI handles all that and more. It's a behemoth of a specification. (whether a particular implementation implements all that stuff correctly is another matter...)

Indeed, there are iSCSI implementations that do multipath and failover.

Both iSCSI and ENBD currently have issues with pending writes during network outages. The current I/O layer fails to report failed writes to fsync and friends.

Quoted 6 lines
Quoted 2 lines

BTW, I'm a big enbd fan :) I like enbd for it's _simplicity_ compared to iSCSI.

Definitely. The iSCSI protocol is more powerful but _much_ more complex than ENBD. I've spent two years working on iSCSI but guess which I use at home..

To be totally fair ENBD/NBD is not SAN nor will it ever become a qualified SAN. I would like to see what happens to your data if you wrap your ethernet around a ballast resistor or even run it near by the associated light fixture, and blink the the power/lights. This is where goofy people run cables in the drop ceilings.

We have almost finalized our initiator to be submitted under OSL/GPL. This will be a full RFC ERL=2 w/ Sync-n-Steering.

I have seen to much GPL code stolen and could do nothing about it.

While the code is not in binary executable form it shall be under OSL only. Only at compile and execution time will it become GPL compliant period. This is designed to extend the copyright holders rights to force anyone who uses the code and changes anything to return to open to the copyright holder period. Additional language may be added to permit not return code to exist under extremely heavy licensing fees to be used to promote OSL projects and assist any GPL holder with litigation fees to protect their rights. To many of us do not have the means to defend our copyrights, this is one way I can see to provide a plausable solution to a dirty problem. This may not be the best answer but it is doable. This will apply some teeth into the license to stop this from happening again.

Like it or hate it, OSL/GPL looks to be the best match out there.

Regards,

Andre Hedrick, CTO & Founder
iSCSI Software Solutions Provider
http://www.PyXTechnologies.com/

#8 lore raw
Quoted 2 lines · Andre Hedrick

We have almost finalized our initiator to be submitted under OSL/GPL. This will be a full RFC ERL=2 w/ Sync-n-Steering.

That's pretty good news.

Also, I tangent and mention that I have been won over WRT OSL: with its more tight "lawyerspeak" and mutual patent defense clauses, I consider OSL to be a "better GPL" license.

I would in fact love to see the Linux kernel relicensed under OSL. I think that would close some "holes" that exist with the GPL, and give us a better legal standing. But relicensing the kernel would be huge political undertaking, and I sure as hell don't have the energy, even if it possible. No idea if Linus, Alan, Andrew, or any of the other major contributors would go for it, either.

Jeff, the radical

#9 lore raw
In reply to Matt Mackall
Quoted 6 lines · Matt Mackall
Quoted 3 lines

Yeah, iSCSI handles all that and more. It's a behemoth of a specification. (whether a particular implementation implements all that stuff correctly is another matter...)

Indeed, there are iSCSI implementations that do multipath and failover.

iSCSI is a transport.
logically, any "multipathing" and "failover" belongs in a layer above it -- typically as a block-layer function -- and not as a transport-layer function.

multipathing belongs elsewhere -- whether it be in MD, LVM, EVMS, DevMapper -- or in a commercial implementation such as Veritas VxDMP, HDS HDLM, EMC PowerPath, ...

Quoted 3 lines

Both iSCSI and ENBD currently have issues with pending writes during network outages. The current I/O layer fails to report failed writes to fsync and friends.

these are not "iSCSI" or "ENBD" issues. these are issues with VFS.

cheers,

lincoln.

#10 lore raw
Quoted 4 lines · Lincoln Dale
Quoted 2 lines

Indeed, there are iSCSI implementations that do multipath and failover.

iSCSI is a transport.
logically, any "multipathing" and "failover" belongs in a layer above it --

"Multipathing" on iSCSI is actually a layer below - network resiliency is handled by routing protocols, the switching fabric etc.

Quoted 4 lines
Quoted 3 lines

Both iSCSI and ENBD currently have issues with pending writes during network outages. The current I/O layer fails to report failed writes to fsync and friends.

these are not "iSCSI" or "ENBD" issues. these are issues with VFS.

Yes, and it is a fairly annoying issue... In particular with ENBD, a partial write could occur at the block device layer. Now try to report that upwards to the write() call. Good luck.

Sincerely,
Lars Marowsky-Brée <[email protected]>

Signature · 4 lines
SuSE Labs - Research & Development, SuSE Linux AG
  
"If anything can go wrong, it will." "Chance favors the prepared (mind)."
  -- Capt. Edward A. Murphy            -- Louis Pasteur
#11 lore raw
Quoted 3 lines · Lars Marowsky-Bree

In particular with ENBD, a partial write could occur at the block device layer. Now try to report that upwards to the write() call. Good luck.

Well you can't, unless it is an O_SYNC or O_DIRECT write...

But yes, for a normal old write() followed by an fsync() the IO error can be lost. We'll fix this for 2.6. I have oxymoron's patches lined up, but they need a couple of quality hours' worth of thinking about yet.

#12 lore raw
In reply to Lincoln Dale
Quoted 11 lines · Lincoln Dale
Quoted 7 lines · Matt Mackall
Quoted 2 lines

Yeah, iSCSI handles all that and more. It's a behemoth of a specification. (whether a particular implementation implements all

that

Quoted 1 line

stuff correctly is another matter...)

Indeed, there are iSCSI implementations that do multipath and failover.

Quoted 6 lines

iSCSI is a transport.
logically, any "multipathing" and "failover" belongs in a layer above it -- typically as a block-layer function -- and not as a transport-layer function.

multipathing belongs elsewhere -- whether it be in MD, LVM, EVMS, DevMapper -- or in a commercial implementation such as Veritas VxDMP, HDS HDLM, EMC PowerPath, ...

I think you will find that most Linux kernel developers agree w/ you :)

That said, iSCSI error recovery can be considered as supporting some of what multipathing and failover accomplish. iSCSI can be shoving bits through multiple TCP connections, or fail over from one TCP connection to another.

Quoted 3 lines
Quoted 3 lines

Both iSCSI and ENBD currently have issues with pending writes during network outages. The current I/O layer fails to report failed writes to fsync and friends.

...not if your iSCSI implementation is up to spec. ;-)

Quoted 1 line

these are not "iSCSI" or "ENBD" issues. these are issues with VFS.

VFS+VM. But, agreed.

Jeff

#13 lore raw
In reply to Lincoln Dale
Quoted 8 lines · Lincoln Dale
Quoted 6 lines · Matt Mackall
Quoted 3 lines

Yeah, iSCSI handles all that and more. It's a behemoth of a specification. (whether a particular implementation implements all that stuff correctly is another matter...)

Indeed, there are iSCSI implementations that do multipath and failover.

Quoted 7 lines

iSCSI is a transport.
logically, any "multipathing" and "failover" belongs in a layer above it -- typically as a block-layer function -- and not as a transport-layer function.

multipathing belongs elsewhere -- whether it be in MD, LVM, EVMS, DevMapper PowerPath, ...

Funny then that I should be talking about Cisco's driver. :P

iSCSI inherently has more interesting reconnect logic than other block devices, so it's fairly trivial to throw in recognition of identical devices discovered on two or more iSCSI targets..

Quoted 5 lines
Quoted 3 lines

Both iSCSI and ENBD currently have issues with pending writes during network outages. The current I/O layer fails to report failed writes to fsync and friends.

these are not "iSCSI" or "ENBD" issues. these are issues with VFS.

Except that the issue simply doesn't show up for anyone else, which is why it hasn't been fixed yet. Patches are in the works, but they need more testing:

http://www.selenic.com/linux/write-error-propagation/

Signature · 1 line
Matt Mackall : http://www.selenic.com : of or relating to the moon

A message that is not in the archive (sent off-list or not yet received): [email protected]

#15 lore raw
Quoted 21 lines · Lincoln Dale
Quoted 6 lines · Matt Mackall
Quoted 5 lines
Quoted 2 lines

Indeed, there are iSCSI implementations that do multipath and failover.

iSCSI is a transport.
logically, any "multipathing" and "failover" belongs in a layer above

it --

Quoted 8 lines
Quoted 4 lines

typically as a block-layer function -- and not as a transport-layer function.

multipathing belongs elsewhere -- whether it be in MD, LVM, EVMS,

DevMapper

Quoted 1 line

PowerPath, ...

Funny then that I should be talking about Cisco's driver. :P

:-)

see my previous email to Jeff. iSCSI as a transport protocol does have a muxing capability -- but its usefulness is somewhat limited (imho).

Quoted 7 lines
Quoted 3 lines

iSCSI inherently has more interesting reconnect logic than other block devices, so it's fairly trivial to throw in recognition of identical devices discovered on two or more iSCSI targets..

what logic do you use to identify "identical devices"? same data reported from SCSI Report_LUNs? or perhaps the same data reported from a SCSI_Inquiry?

Sorry, can't remember.

Quoted 2 lines

does one now need to add logic into the kernel to provide some multipathing for HDS disks?

No, most of it was done in userspace.

Quoted 10 lines
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

Both iSCSI and ENBD currently have issues with pending writes during network outages. The current I/O layer fails to report failed writes to fsync and friends.

these are not "iSCSI" or "ENBD" issues. these are issues with VFS.

Except that the issue simply doesn't show up for anyone else, which is why it hasn't been fixed yet. Patches are in the works, but they need more testing:

Quoted 5 lines
Quoted 1 line

http://www.selenic.com/linux/write-error-propagation/

oh, but it does show up for other people. it may be that the issue doesn't show up at fsync() time, but rather at close() time, or perhaps neither of those!

Write errors basically don't happen for people who have attached storage unless their drives die. Which is why the fact that the pagecache completely ignores I/O errors has gone unnoticed for years..

Quoted 3 lines

code looks interesting. i'll take a look.
hmm, must find out a way to intentionally introduce errors now and see what happens!

We stumbled on it by pulling cables to make failover happen.

Signature · 1 line
Matt Mackall : http://www.selenic.com : of or relating to the moon
#16 lore raw
In reply to Peter T. Breuer

Hi!

Quoted 4 lines

9) it drops into a mode where it md5sums both ends and skips writes of equal blocks, if that's faster. It moves in and out of this mode automatically. This helps RAID resyncs (2* overspeed is common on 100BT nets, that is 20MB/s.).

Great way to find md5 collisions, I guess
:-).
Pavel

Signature · 2 lines
				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...
#17 lore raw

"A month of sundays ago Pavel Machek wrote:"

Quoted 9 lines

Hi!

Quoted 4 lines

9) it drops into a mode where it md5sums both ends and skips writes of equal blocks, if that's faster. It moves in and out of this mode automatically. This helps RAID resyncs (2* overspeed is common on 100BT nets, that is 20MB/s.).

Great way to find md5 collisions, I guess
:-).

Don't worry, I'm not planning on claiming the Turing medal! Or living for the lifetime of the universe .. :-(.

Peter