Skip to contentSkip to conversation

solid state drive access and context switching

DISCUSSION Chris Friesen 6 participants 16 messages Dec 3, 2007 – Dec 6, 2007 Block layer

lore
Share & tools
lore
#0 lore raw

Over on comp.os.linux.development.system someone asked an interesting question, and I thought I'd mention it here.

Given a fast low-latency solid state drive, would it ever be beneficial to simply wait in the kernel for synchronous read/write calls to complete? The idea is that you could avoid at least two task context switches, and if the data access can be completed at less cost than those context switches it could be an overall win.

Has anyone played with this concept?

Chris

#1 lore raw
Quoted 5 lines

Given a fast low-latency solid state drive, would it ever be beneficial to simply wait in the kernel for synchronous read/write calls to complete? The idea is that you could avoid at least two task context switches, and if the data access can be completed at less cost than those context switches it could be an overall win.

In certain situations theoretically yes, the kernel is better off continuing to poll than switching to the idle thread. You can do this to some extent in a driver already today - just poll rather than sleeping but respsect the reschedule hints and don't do it with irqs masked.

Quoted 1 line

Has anyone played with this concept?

For things like SATA based devices they aren't that fast yet.

Alan

#2 lore raw
Quoted 3 lines
Quoted 1 line

Has anyone played with this concept?

For things like SATA based devices they aren't that fast yet.

What is fast enough?

As I understand the basic memory technology, the hard limit is in the 100's of microseconds range for latency. SATA adds something to that.
I'd be surprised to see latencies on SATA SSD's as measured at the OS
level to get below 1 millisecond.

What happens we start placing NAND technology in lower latency, higher bandwidth buses? I'm guessing we'll get down to that 100's of microseconds level and an order of magnitude higher bandwidth than SATA. Is that fast enough to warrant this more synchronous IO?

Magnetic drives have latencies ~10 milliseconds, current SSD's are an order of magnitude better (~1 millisecond), new interfaces and refinements could theoretically get us down one more (~100 microsecond). I'm guessing the current block driver subsystem would negate a lot of that latency gain. Am I wrong?

BTW - This trend toward faster, lower latency busses is marching forward. 2 examples; the ioDrive from Fusion IO, Micron's RAM-module like SSD concept.

#3 lore raw
Quoted 2 lines

microseconds level and an order of magnitude higher bandwidth than SATA. Is that fast enough to warrant this more synchronous IO?

See the mtd layer.

Quoted 3 lines

BTW - This trend toward faster, lower latency busses is marching forward. 2 examples; the ioDrive from Fusion IO, Micron's RAM-module like SSD concept.

Very much so but we can do quite a bit in 10,000 processor cycles ...

Alan

#12 lore raw
In reply to Jared Hulbert
Quoted 4 lines · Jared Hulbert

Magnetic drives have latencies ~10 milliseconds, current SSD's are an order of magnitude better (~1 millisecond), new interfaces and refinements could theoretically get us down one more (~100 microsecond).

They've already done already better than that. Here's a solid state drive with a claimed 20 microsecond access time:

http://www.curtisssd.com/products/drives/hyperxclr

Chris

#14 lore raw
In reply to Alan Cox
Quoted 1 line · Alan Cox

For things like SATA based devices they aren't that fast yet.

You forget the Gigabyte i-RAM.

For others: the i-RAM is a SATA-based device that plugs into a PCI slot on your motherboard (for power), providing RAM+battery backup as fast as your SATA bus and DIMMs will go.

Jeff

#15 lore raw
Quoted 8 lines · On Tue, 04 Dec 2007 15:52:20 -0500 Jeff Garzik <[email protected]> wrote:
Quoted 1 line · Alan Cox

For things like SATA based devices they aren't that fast yet.

You forget the Gigabyte i-RAM.

For others: the i-RAM is a SATA-based device that plugs into a PCI slot on your motherboard (for power), providing RAM+battery backup as fast as your SATA bus and DIMMs will go.

Actually even allowing for the iRAM the SATA stuff is way too slow to be worth using synchronously. The latency is a killer.

#13 lore raw
In reply to Chris Friesen
Quoted 7 lines
Quoted 2 lines

refinements could theoretically get us down one more (~100 microsecond).

They've already done already better than that. Here's a solid state drive with a claimed 20 microsecond access time:

http://www.curtisssd.com/products/drives/hyperxclr

Right. That looks to be RAM based, which means $$$$ compared to NAND, so that's not going to breakout of a server niche. I imagine the latency is the device latency not the system latency. By the time you send the request through the fibrechannel stack and get the block back it's gonna be much closer to 100 microseconds. It's that OS visible latency that you've got to design to.

#4 lore raw
In reply to Alan Cox
Quoted 4 lines
Quoted 2 lines

microseconds level and an order of magnitude higher bandwidth than SATA. Is that fast enough to warrant this more synchronous IO?

See the mtd layer.

Right. The trend is to hide the nastiness of NAND technology changes behind controllers. In general I think this is a good thing. Basically the changes in ECC and reliability change very rapidly in this technology. Having custom controller hardware to handle this is faster than handling it in software and makes for a nice modular interface. We don't rewrite our SATA drivers and filesystem everything the magnetic media switches to a new recording scheme, we just plug it in. SSD's are going to be like that even if they aren't SATA. However, the MTD layer is more about managing the chips themselves, which is what the controllers are for.

Maybe I'm missing something but I don't see it. We want a block interface for these devices, we just need a faster slimmer interface. Maybe a new mtdblock interface that doesn't do erase would be the place for?

Quoted 8 lines
Quoted 3 lines

BTW - This trend toward faster, lower latency busses is marching forward. 2 examples; the ioDrive from Fusion IO, Micron's RAM-module like SSD concept.

Very much so but we can do quite a bit in 10,000 processor cycles ...

Alan

#5 lore raw
Quoted 5 lines · Jared Hulbert

Maybe I'm missing something but I don't see it. We want a block interface for these devices, we just need a faster slimmer interface. Maybe a new mtdblock interface that doesn't do erase would be the place for?

Doesn't do erase? MTD has to learn almost all tricks from the block layer, as devices are becoming high-latency high-bandwidth, compared to what MTD was designed for. In order to get any decent performance, we need asynchronous operations, request queues and caching.

The only useful advantage MTD does have over block devices is an _explicit_ erase operation. Did you mean "doesn't do _implicit_ erase".

Jörn

Signature · 2 lines
It's just what we asked for, but not what we want!
-- anonymous
#7 lore raw
In reply to Jared Hulbert
Quoted 2 lines

Right. The trend is to hide the nastiness of NAND technology changes behind controllers. In general I think this is a good thing.

You miss the point - any controller you hide it behind almost inevitably adds enough latency you don't want to use it synchronously.

Alan

#6 lore raw
In reply to Jörn Engel
Quoted 5 lines
Quoted 4 lines

Maybe I'm missing something but I don't see it. We want a block interface for these devices, we just need a faster slimmer interface. Maybe a new mtdblock interface that doesn't do erase would be the place for?

Quoted 7 lines

Doesn't do erase? MTD has to learn almost all tricks from the block layer, as devices are becoming high-latency high-bandwidth, compared to what MTD was designed for. In order to get any decent performance, we need asynchronous operations, request queues and caching.

The only useful advantage MTD does have over block devices is an _explicit_ erase operation. Did you mean "doesn't do _implicit_ erase".

You're right. That the point I was trying to make, albeit badly, MTD isn't the place for this. The fact that more and more of what the MTD is being used for looks a lot like the block layer is a whole different discussion.

#8 lore raw
In reply to Alan Cox
Quoted 5 lines · Alan Cox
Quoted 2 lines

Right. The trend is to hide the nastiness of NAND technology changes behind controllers. In general I think this is a good thing.

You miss the point - any controller you hide it behind almost inevitably adds enough latency you don't want to use it synchronously.

I think I get it. We keep saying that it's the latency is too high. I agree that most technologies out there have latencies that are too high. Again I ask the question, what latencies do we have to hit before the sync options become worth it?

#9 lore raw
Quoted 11 lines · On Tue, 4 Dec 2007 16:08:07 -0800 "Jared Hulbert" <[email protected]> wrote:
Quoted 5 lines · Alan Cox
Quoted 2 lines

Right. The trend is to hide the nastiness of NAND technology changes behind controllers. In general I think this is a good thing.

You miss the point - any controller you hide it behind almost inevitably adds enough latency you don't want to use it synchronously.

I think I get it. We keep saying that it's the latency is too high. I agree that most technologies out there have latencies that are too high. Again I ask the question, what latencies do we have to hit before the sync options become worth it?

Probably about 1000 clocks but its always going to depend upon the workload and whether any other work can be done usefully.

Alan

#10 lore raw
Quoted 2 lines

Probably about 1000 clocks but its always going to depend upon the workload and whether any other work can be done usefully.

Yeah. Sounds right, in the microsecond range. Be interesting to see data.

Anybody have ideas on what kind of experiments could confirm this estimate is right?

#11 lore raw

Hi,

Quoted 7 lines · Jared Hulbert
Quoted 2 lines

Probably about 1000 clocks but its always going to depend upon the workload and whether any other work can be done usefully.

Yeah. Sounds right, in the microsecond range. Be interesting to see data.

Anybody have ideas on what kind of experiments could confirm this estimate is right?

Is it the right place to write synchronously?
Now only concern the SATA.

Thank you,
Kyungmin Park

block/ll_rw_blk.c +7 -0
index 3b927be..cce0618 100644
@@ -3221,6 +3221,13 @@ static inline void __generic_make_request(struct bio *bio
        if (bio_check_eod(bio, nr_sectors))
                goto end_io;
 
+#if 1
+       /* FIXME simple hack */
+       if (MAJOR(bio->bi_bdev->bd_dev) == 8 && bio_data_dir(bio) == WRITE) {
+               /* WRITE_SYNC */
+               bio->bi_rw |= (1 << BIO_RW_SYNC);
+       }
+#endif
        /*
         * Resolve the mapping until finished. (drivers are
         * still free to implement/resolve their own stacking