virtio-spec: document block CMD and FLUSH

Quoted 2 lines · Michael S. Tsirkin

I took a stub at documenting CMD and FLUSH request types in virtio block.

Any comments?

Michael S. Tsirkin

author maintainer

2010-04-28 15:52 UTC #3 lore raw

Quoted 5 lines · Michael S. Tsirkin

Quoted 2 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block.
Any comments?

Rusty?

Michael S. Tsirkin

author maintainer

2010-04-28 15:56 UTC #4 lore raw

Quoted 5 lines · Michael S. Tsirkin

Quoted 2 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block.
Any comments?

Rusty?

Jamie Lokier

2010-04-20 01:46 UTC #5 lore raw

Quoted 299 lines · Michael S. Tsirkin

I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?

I note that the interface seems full of warts to me, this might be a first step to cleaning them.

One issue I struggled with especially is how type
field mixes bits and non-bit values. I ended up
simply defining all legal values, so that we have
CMD = 2, CMD_OUT = 3 and so on.

I also avoided instroducing inhdr/outhdr structures that virtio blk driver in linux uses, I was concerned that nesting tables will confuse the reader.

Comments welcome.

Signed-off-by Michael S. Tsirkin <[email protected]>

virtio-spec.lyx +207 -7

index d16104a..ed35893 100644
@@ -67,7 +67,11 @@ IBM Corporation
 \end_layout
 
 \begin_layout Standard
+
+\change_deleted 0 1266531118
 FIXME: virtio block scsi passthrough section
+\change_unchanged
+
 \end_layout
 
 \begin_layout Standard

@@ -4376,7 +4380,7 @@ struct virtio_net_ctrl_mac {
 The device can filter incoming packets by any number of destination MAC
  addresses.
 \begin_inset Foot
-status open
+status collapsed
 
 \begin_layout Plain Layout
 Since there are no guarentees, it can use a hash filter orsilently switch

@@ -4549,6 +4553,22 @@ blk_size
 \end_inset
 
 .
+\change_inserted 0 1266444580
+
+\end_layout
+
+\begin_layout Description
+
+\change_inserted 0 1266471229
+VIRTIO_BLK_F_SCSI (7) Device supports scsi packet commands.
+\end_layout
+
+\begin_layout Description
+
+\change_inserted 0 1266444605
+VIRTIO_BLK_F_FLUSH (9) Cache flush command support.
+\change_unchanged
+
 \end_layout
 
 \begin_layout Description

@@ -4700,17 +4720,25 @@ struct virtio_blk_req {
 
 \begin_layout Plain Layout
 
+\change_deleted 0 1266472188
+
 #define VIRTIO_BLK_T_IN          0
 \end_layout
 
 \begin_layout Plain Layout
 
+\change_deleted 0 1266472188
+
 #define VIRTIO_BLK_T_OUT         1
 \end_layout
 
 \begin_layout Plain Layout
 
+\change_deleted 0 1266472188
+
 #define VIRTIO_BLK_T_BARRIER	 0x80000000
+\change_unchanged
+
 \end_layout
 
 \begin_layout Plain Layout

@@ -4735,11 +4763,15 @@ struct virtio_blk_req {
 
 \begin_layout Plain Layout
 
+\change_deleted 0 1266472204
+
 #define VIRTIO_BLK_S_OK        0
 \end_layout
 
 \begin_layout Plain Layout
 
+\change_deleted 0 1266472204
+
 #define VIRTIO_BLK_S_IOERR     1
 \end_layout

@@ -4759,32 +4791,481 @@ struct virtio_blk_req {
 \end_layout
 
 \begin_layout Standard
-The type of the request is either a read (VIRTIO_BLK_T_IN) or a write (VIRTIO_BL
-K_T_OUT); the high bit indicates that this request acts as a barrier and
- that all preceeding requests must be complete before this one, and all
- following requests must not be started until this is complete.
+
+\change_inserted 0 1266472490
+If the device has VIRTIO_BLK_F_SCSI feature, it can also support scsi packet
+ command requests, each of these requests is of form:
+\begin_inset listings
+inline false
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472395
+
+struct virtio_scsi_pc_req {
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472375
+
+	u32 type;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472375
+
+	u32 ioprio;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266474298
+
+	u64 sector;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266474308
+
+    char cmd[];
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266505809
+
+	char data[][512];
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266505825
+
+#define SCSI_SENSE_BUFFERSIZE   96
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266505848
+
+    u8 sense[SCSI_SENSE_BUFFERSIZE];
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472969
+
+    u32 errors;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472979
+
+    u32 data_len;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472984
+
+    u32 sense_len;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472987
+
+    u32 residual;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472375
+
+	u8 status;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472375
+
+};
+\end_layout
+
+\end_inset
+
+
+\change_unchanged
+
 \end_layout
 
 \begin_layout Standard
-The ioprio field is a hint about the relative priorities of requests to
- the device: higher numbers indicate more important requests.
+The 
+\emph on
+type
+\emph default
+ of the request is either a read (VIRTIO_BLK_T_IN)
+\change_inserted 0 1266495815
+,
+\change_unchanged
+ 
+\change_deleted 0 1266495817
+or
+\change_unchanged
+ a write (VIRTIO_BLK_T_OUT)
+\change_inserted 0 1266497316
+, a scsi packet command (VIRTIO_BLK_T_SCSI_CMD or VIRTIO_BLK_T_SCSI_CMD_OUT
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266497390
+the SCSI_CMD and SCSI_CMD_OUT types are equivalent, the device does not
+ distinguish between them
+\change_unchanged
+
+\end_layout
+
+\end_inset
+
+) or a flush (VIRTIO_BLK_T_FLUSH or VIRTIO_BLK_T_FLUSH_OUT
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266497402
+the FLUSH and FLUSH_OUT types are equivalent, the device does not distinguish
+ between them
+\change_unchanged
+
+\end_layout
+
+\end_inset
+
+)
+\change_deleted 0 1266503753
+;
+\change_inserted 0 1266503758
+.
+
+\change_unchanged
+ 
+\change_inserted 0 1266497301
+If the device has VIRTIO_BLK_F_BARRIER feature
+\begin_inset space ~
+\end_inset
+
+
+\change_unchanged
+the high bit
+\change_inserted 0 1266497301
 
+ (VIRTIO_BLK_T_BARRIER)
+\change_unchanged

virtio-spec.lyx struct virtio_blk_req { · line 4981

Quoted 10 lines

virtio-spec.lyx +10 -0

+ indicates that this request acts as a barrier and that all preceeding requests
+ must be complete before this one, and all following requests must not be
+ started until this is complete.
+
+\change_inserted 0 1266504385
+ Note that a barrier does not flush caches in the underlying backend device
+ in host, and thus does not serve as data consistency guarantee.
+ Driver must use FLUSH request to flush the host cache.
+\change_unchanged
+

Does this mean that virtio-blk supports all three combinations?

1. FLUSH that isn't a barrier
2. FLUSH that is also a barrier
3. Barrier that is not a flush

1 is good for fsync-like operations;
2 is good for journalling-like ordered operations.
3 sounds like it doesn't mean a lot as the host cache provides no guarantees and has no ordering facility that can be used.

-- Jamie

Jamie Lokier

2010-04-20 01:46 UTC #6 lore raw

Quoted 299 lines · Michael S. Tsirkin

I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?

I note that the interface seems full of warts to me, this might be a first step to cleaning them.

One issue I struggled with especially is how type
field mixes bits and non-bit values. I ended up
simply defining all legal values, so that we have
CMD = 2, CMD_OUT = 3 and so on.

I also avoided instroducing inhdr/outhdr structures that virtio blk driver in linux uses, I was concerned that nesting tables will confuse the reader.

Comments welcome.

Signed-off-by Michael S. Tsirkin <[email protected]>

virtio-spec.lyx +207 -7

index d16104a..ed35893 100644
@@ -67,7 +67,11 @@ IBM Corporation
 \end_layout
 
 \begin_layout Standard
+
+\change_deleted 0 1266531118
 FIXME: virtio block scsi passthrough section
+\change_unchanged
+
 \end_layout
 
 \begin_layout Standard

@@ -4376,7 +4380,7 @@ struct virtio_net_ctrl_mac {
 The device can filter incoming packets by any number of destination MAC
  addresses.
 \begin_inset Foot
-status open
+status collapsed
 
 \begin_layout Plain Layout
 Since there are no guarentees, it can use a hash filter orsilently switch

@@ -4549,6 +4553,22 @@ blk_size
 \end_inset
 
 .
+\change_inserted 0 1266444580
+
+\end_layout
+
+\begin_layout Description
+
+\change_inserted 0 1266471229
+VIRTIO_BLK_F_SCSI (7) Device supports scsi packet commands.
+\end_layout
+
+\begin_layout Description
+
+\change_inserted 0 1266444605
+VIRTIO_BLK_F_FLUSH (9) Cache flush command support.
+\change_unchanged
+
 \end_layout
 
 \begin_layout Description

@@ -4700,17 +4720,25 @@ struct virtio_blk_req {
 
 \begin_layout Plain Layout
 
+\change_deleted 0 1266472188
+
 #define VIRTIO_BLK_T_IN          0
 \end_layout
 
 \begin_layout Plain Layout
 
+\change_deleted 0 1266472188
+
 #define VIRTIO_BLK_T_OUT         1
 \end_layout
 
 \begin_layout Plain Layout
 
+\change_deleted 0 1266472188
+
 #define VIRTIO_BLK_T_BARRIER	 0x80000000
+\change_unchanged
+
 \end_layout
 
 \begin_layout Plain Layout

@@ -4735,11 +4763,15 @@ struct virtio_blk_req {
 
 \begin_layout Plain Layout
 
+\change_deleted 0 1266472204
+
 #define VIRTIO_BLK_S_OK        0
 \end_layout
 
 \begin_layout Plain Layout
 
+\change_deleted 0 1266472204
+
 #define VIRTIO_BLK_S_IOERR     1
 \end_layout

@@ -4759,32 +4791,481 @@ struct virtio_blk_req {
 \end_layout
 
 \begin_layout Standard
-The type of the request is either a read (VIRTIO_BLK_T_IN) or a write (VIRTIO_BL
-K_T_OUT); the high bit indicates that this request acts as a barrier and
- that all preceeding requests must be complete before this one, and all
- following requests must not be started until this is complete.
+
+\change_inserted 0 1266472490
+If the device has VIRTIO_BLK_F_SCSI feature, it can also support scsi packet
+ command requests, each of these requests is of form:
+\begin_inset listings
+inline false
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472395
+
+struct virtio_scsi_pc_req {
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472375
+
+	u32 type;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472375
+
+	u32 ioprio;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266474298
+
+	u64 sector;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266474308
+
+    char cmd[];
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266505809
+
+	char data[][512];
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266505825
+
+#define SCSI_SENSE_BUFFERSIZE   96
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266505848
+
+    u8 sense[SCSI_SENSE_BUFFERSIZE];
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472969
+
+    u32 errors;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472979
+
+    u32 data_len;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472984
+
+    u32 sense_len;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472987
+
+    u32 residual;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472375
+
+	u8 status;
+\end_layout
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266472375
+
+};
+\end_layout
+
+\end_inset
+
+
+\change_unchanged
+
 \end_layout
 
 \begin_layout Standard
-The ioprio field is a hint about the relative priorities of requests to
- the device: higher numbers indicate more important requests.
+The 
+\emph on
+type
+\emph default
+ of the request is either a read (VIRTIO_BLK_T_IN)
+\change_inserted 0 1266495815
+,
+\change_unchanged
+ 
+\change_deleted 0 1266495817
+or
+\change_unchanged
+ a write (VIRTIO_BLK_T_OUT)
+\change_inserted 0 1266497316
+, a scsi packet command (VIRTIO_BLK_T_SCSI_CMD or VIRTIO_BLK_T_SCSI_CMD_OUT
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266497390
+the SCSI_CMD and SCSI_CMD_OUT types are equivalent, the device does not
+ distinguish between them
+\change_unchanged
+
+\end_layout
+
+\end_inset
+
+) or a flush (VIRTIO_BLK_T_FLUSH or VIRTIO_BLK_T_FLUSH_OUT
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+
+\change_inserted 0 1266497402
+the FLUSH and FLUSH_OUT types are equivalent, the device does not distinguish
+ between them
+\change_unchanged
+
+\end_layout
+
+\end_inset
+
+)
+\change_deleted 0 1266503753
+;
+\change_inserted 0 1266503758
+.
+
+\change_unchanged
+ 
+\change_inserted 0 1266497301
+If the device has VIRTIO_BLK_F_BARRIER feature
+\begin_inset space ~
+\end_inset
+
+
+\change_unchanged
+the high bit
+\change_inserted 0 1266497301
 
+ (VIRTIO_BLK_T_BARRIER)
+\change_unchanged

virtio-spec.lyx struct virtio_blk_req { · line 4981

Quoted 10 lines

virtio-spec.lyx +10 -0

+ indicates that this request acts as a barrier and that all preceeding requests
+ must be complete before this one, and all following requests must not be
+ started until this is complete.
+
+\change_inserted 0 1266504385
+ Note that a barrier does not flush caches in the underlying backend device
+ in host, and thus does not serve as data consistency guarantee.
+ Driver must use FLUSH request to flush the host cache.
+\change_unchanged
+

Does this mean that virtio-blk supports all three combinations?

1. FLUSH that isn't a barrier
2. FLUSH that is also a barrier
3. Barrier that is not a flush

-- Jamie

Paul Brook

2010-04-20 13:22 UTC #7 lore raw

Quoted 10 lines

Does this mean that virtio-blk supports all three combinations?
1. FLUSH that isn't a barrier
2. FLUSH that is also a barrier
3. Barrier that is not a flush
1 is good for fsync-like operations;
2 is good for journalling-like ordered operations.
3 sounds like it doesn't mean a lot as the host cache provides no guarantees and has no ordering facility that can be used.

(3) allows the guest to queue overlapping transfers with well defined results. I have no idea how useful this is in practice, but it's certainly plausible.

Paul

Paul Brook

2010-04-20 13:23 UTC #8 lore raw

In reply to Jamie Lokier

Quoted 10 lines

Does this mean that virtio-blk supports all three combinations?
1. FLUSH that isn't a barrier
2. FLUSH that is also a barrier
3. Barrier that is not a flush
1 is good for fsync-like operations;
2 is good for journalling-like ordered operations.
3 sounds like it doesn't mean a lot as the host cache provides no guarantees and has no ordering facility that can be used.

(3) allows the guest to queue overlapping transfers with well defined results. I have no idea how useful this is in practice, but it's certainly plausible.

Paul

Michael S. Tsirkin

author maintainer

2010-04-21 10:39 UTC #9 lore raw

Quoted 11 lines · Paul Brook

Quoted 10 lines
Does this mean that virtio-blk supports all three combinations?
1. FLUSH that isn't a barrier
2. FLUSH that is also a barrier
3. Barrier that is not a flush
1 is good for fsync-like operations;
2 is good for journalling-like ordered operations.
3 sounds like it doesn't mean a lot as the host cache provides no guarantees and has no ordering facility that can be used.

Quoted 4 lines

(3) allows the guest to queue overlapping transfers with well defined results. I have no idea how useful this is in practice, but it's certainly plausible.
Paul

In theory, yes.
At the moment, qemu only implements FLUSH and lguest only implements barrier without FLUSH.

If you think it's useful, maybe start by using FLUSH+barrier in linux guest driver, that'd demonstrate how it's used.

Signature · 1 line

MST

Michael S. Tsirkin

author maintainer

2010-04-21 10:44 UTC #10 lore raw

In reply to Paul Brook

Quoted 11 lines · Paul Brook

Quoted 10 lines
Does this mean that virtio-blk supports all three combinations?
1. FLUSH that isn't a barrier
2. FLUSH that is also a barrier
3. Barrier that is not a flush
1 is good for fsync-like operations;
2 is good for journalling-like ordered operations.
3 sounds like it doesn't mean a lot as the host cache provides no guarantees and has no ordering facility that can be used.

Quoted 4 lines

(3) allows the guest to queue overlapping transfers with well defined results. I have no idea how useful this is in practice, but it's certainly plausible.
Paul

In theory, yes.
At the moment, qemu only implements FLUSH and lguest only implements barrier without FLUSH.

If you think it's useful, maybe start by using FLUSH+barrier in linux guest driver, that'd demonstrate how it's used.

Signature · 1 line

MST

Christoph Hellwig

2010-05-04 18:56 UTC #11 lore raw

In reply to Jamie Lokier

Quoted 10 lines · Jamie Lokier

Does this mean that virtio-blk supports all three combinations?
1. FLUSH that isn't a barrier
2. FLUSH that is also a barrier
3. Barrier that is not a flush
1 is good for fsync-like operations;
2 is good for journalling-like ordered operations.
3 sounds like it doesn't mean a lot as the host cache provides no guarantees and has no ordering facility that can be used.

No. The Linux virtio_blk guest driver either supports data integrity by using FLUSH or can send down BARRIER requests which aren't much help at all. Qemu only implements FLUSH anyway.

Michael S. Tsirkin

author maintainer

2010-05-04 19:01 UTC #12 lore raw

Quoted 7 lines · Christoph Hellwig

Quoted 6 lines · Jamie Lokier
Does this mean that virtio-blk supports all three combinations?
1. FLUSH that isn't a barrier
2. FLUSH that is also a barrier
3. Barrier that is not a flush

Quoted 8 lines

Quoted 4 lines
1 is good for fsync-like operations;
2 is good for journalling-like ordered operations.
3 sounds like it doesn't mean a lot as the host cache provides no guarantees and has no ordering facility that can be used.
No. The Linux virtio_blk guest driver either supports data integrity by using FLUSH or can send down BARRIER requests which aren't much help at all.

It seems we use BARRIER when we get REQ_HARDBARRIER, right? What does the REQ_HARDBARRIER flag in request mean and when is it set?

Quoted 1 line

Qemu only implements FLUSH anyway.

Michael S. Tsirkin

author maintainer

2010-05-04 19:05 UTC #13 lore raw

Quoted 7 lines · Christoph Hellwig

Quoted 6 lines · Jamie Lokier
Does this mean that virtio-blk supports all three combinations?
1. FLUSH that isn't a barrier
2. FLUSH that is also a barrier
3. Barrier that is not a flush

Quoted 8 lines

Quoted 4 lines
1 is good for fsync-like operations;
2 is good for journalling-like ordered operations.
3 sounds like it doesn't mean a lot as the host cache provides no guarantees and has no ordering facility that can be used.
No. The Linux virtio_blk guest driver either supports data integrity by using FLUSH or can send down BARRIER requests which aren't much help at all.

It seems we use BARRIER when we get REQ_HARDBARRIER, right? What does the REQ_HARDBARRIER flag in request mean and when is it set?

Quoted 1 line

Qemu only implements FLUSH anyway.

Rusty Russell

2010-05-04 04:38 UTC #14 lore raw

Quoted 5 lines · Michael S. Tsirkin

I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.

ISTR Christoph had withdrawn some patches in this area, and was waiting for him to resubmit?

I've given up on figuring out the block device. What seem to me to be sane semantics along the lines of memory barriers are foreign to disk people: they want (and depend on) flushing everywhere.

For example, tdb transactions do not require a flush, they only require what I would call a barrier: that prior data be written out before any future data. Surely that would be more efficient in general than a flush! In fact, TDB wants only writes to *that file* (and metadata) written out first; it has no ordering issues with other I/O on the same device.

A generic I/O interface would allow you to specify "this request depends on these outstanding requests" and leave it at that. It might have some sync flush command for dumb applications and OSes. The userspace API might be not be as precise and only allow such a barrier against all prior writes on this fd.

ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...

Cheers,
Rusty.

Rusty Russell

2010-05-04 04:38 UTC #15 lore raw

Quoted 5 lines · Michael S. Tsirkin

I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.

ISTR Christoph had withdrawn some patches in this area, and was waiting for him to resubmit?

I've given up on figuring out the block device. What seem to me to be sane semantics along the lines of memory barriers are foreign to disk people: they want (and depend on) flushing everywhere.

ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...

Cheers,
Rusty.

Stefan Hajnoczi

2010-05-04 06:56 UTC #16 lore raw

A userspace barrier API would be very useful instead of doing fsync when only ordering is required. I'd like to follow that discussion too.

Stefan

On 4 May 2010 05:39, "Rusty Russell" <[email protected]> wrote:

Quoted 1 line · Michael S. Tsirkin

I took a stub at documenting CMD and FLU...

ISTR Christoph had withdrawn some patches in this area, and was waiting for him to resubmit?

I've given up on figuring out the block device. What seem to me to be sane semantics along the lines of memory barriers are foreign to disk people:
they
want (and depend on) flushing everywhere.

For example, tdb transactions do not require a flush, they only require what I would call a barrier: that prior data be written out before any future data.
Surely that would be more efficient in general than a flush! In fact, TDB wants only writes to *that file* (and metadata) written out first; it has no ordering issues with other I/O on the same device.

A generic I/O interface would allow you to specify "this request depends on these
outstanding requests" and leave it at that. It might have some sync flush command for dumb applications and OSes. The userspace API might be not be as
precise and only allow such a barrier against all prior writes on this fd.

ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...

Cheers,
Rusty.

_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization

Avi Kivity

2010-05-04 08:34 UTC #17 lore raw

Quoted 15 lines · Rusty Russell

Quoted 6 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.
ISTR Christoph had withdrawn some patches in this area, and was waiting for him to resubmit?
I've given up on figuring out the block device. What seem to me to be sane semantics along the lines of memory barriers are foreign to disk people: they want (and depend on) flushing everywhere.

Quoted 6 lines

For example, tdb transactions do not require a flush, they only require what I would call a barrier: that prior data be written out before any future data. Surely that would be more efficient in general than a flush! In fact, TDB wants only writes to *that file* (and metadata) written out first; it has no ordering issues with other I/O on the same device.

I think that's SCSI ordered tags.

Quoted 5 lines

A generic I/O interface would allow you to specify "this request depends on these outstanding requests" and leave it at that. It might have some sync flush command for dumb applications and OSes. The userspace API might be not be as precise and only allow such a barrier against all prior writes on this fd.

Depends on all previous requests, and will commit before all following requests. ie a full barrier.

Quoted 3 lines

ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...

I'd love to see TCQ exposed to user space.

Signature · 1 line

error compiling committee.c: too many arguments to function

Avi Kivity

2010-05-04 08:34 UTC #18 lore raw

Quoted 15 lines · Rusty Russell

Quoted 6 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.
ISTR Christoph had withdrawn some patches in this area, and was waiting for him to resubmit?
I've given up on figuring out the block device. What seem to me to be sane semantics along the lines of memory barriers are foreign to disk people: they want (and depend on) flushing everywhere.

Quoted 6 lines

For example, tdb transactions do not require a flush, they only require what I would call a barrier: that prior data be written out before any future data. Surely that would be more efficient in general than a flush! In fact, TDB wants only writes to *that file* (and metadata) written out first; it has no ordering issues with other I/O on the same device.

I think that's SCSI ordered tags.

Quoted 5 lines

A generic I/O interface would allow you to specify "this request depends on these outstanding requests" and leave it at that. It might have some sync flush command for dumb applications and OSes. The userspace API might be not be as precise and only allow such a barrier against all prior writes on this fd.

Depends on all previous requests, and will commit before all following requests. ie a full barrier.

Quoted 3 lines

ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...

I'd love to see TCQ exposed to user space.

Signature · 1 line

error compiling committee.c: too many arguments to function

Jens Axboe

2010-05-04 08:41 UTC #19 lore raw

Quoted 20 lines · Rusty Russell

Quoted 5 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.
ISTR Christoph had withdrawn some patches in this area, and was waiting for him to resubmit?
I've given up on figuring out the block device. What seem to me to be sane semantics along the lines of memory barriers are foreign to disk people: they want (and depend on) flushing everywhere.
For example, tdb transactions do not require a flush, they only require what I would call a barrier: that prior data be written out before any future data. Surely that would be more efficient in general than a flush! In fact, TDB wants only writes to *that file* (and metadata) written out first; it has no ordering issues with other I/O on the same device.

Quoted 7 lines

A generic I/O interface would allow you to specify "this request depends on these outstanding requests" and leave it at that. It might have some sync flush command for dumb applications and OSes. The userspace API might be not be as precise and only allow such a barrier against all prior writes on this fd.
ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...

It would be nice to have a more fuller API for this, but the reality is that only the flush approach is really workable. Even just strict ordering of requests could only be supported on SCSI, and even there the kernel still lacks proper guarantees on error handling to prevent reordering there.

Signature · 1 line

Jens Axboe

Jens Axboe

2010-05-04 08:41 UTC #20 lore raw

Quoted 20 lines · Rusty Russell

Quoted 5 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.
ISTR Christoph had withdrawn some patches in this area, and was waiting for him to resubmit?
I've given up on figuring out the block device. What seem to me to be sane semantics along the lines of memory barriers are foreign to disk people: they want (and depend on) flushing everywhere.
For example, tdb transactions do not require a flush, they only require what I would call a barrier: that prior data be written out before any future data. Surely that would be more efficient in general than a flush! In fact, TDB wants only writes to *that file* (and metadata) written out first; it has no ordering issues with other I/O on the same device.

Quoted 7 lines

A generic I/O interface would allow you to specify "this request depends on these outstanding requests" and leave it at that. It might have some sync flush command for dumb applications and OSes. The userspace API might be not be as precise and only allow such a barrier against all prior writes on this fd.
ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...

Signature · 1 line

Jens Axboe

Jamie Lokier

2010-05-04 20:17 UTC #21 lore raw

Quoted 9 lines · Jens Axboe

Quoted 2 lines · Rusty Russell
ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...
It would be nice to have a more fuller API for this, but the reality is that only the flush approach is really workable. Even just strict ordering of requests could only be supported on SCSI, and even there the kernel still lacks proper guarantees on error handling to prevent reordering there.

There's a few I/O scheduling differences that might be useful:

1. The I/O scheduler could freely move WRITEs before a FLUSH but not before a BARRIER. That might be useful for time-critical WRITEs, and those issued by high I/O priority.

2. The I/O scheduler could move WRITEs after a FLUSH if the FLUSH is only for data belonging to a particular file (e.g. fdatasync with no file size change, even on btrfs if O_DIRECT was used for the writes being committed). That would entail tagging FLUSHes and WRITEs with a fs-specific identifier (such as inode number), opaque to the scheduler which only checks equality.

3. By delaying FLUSHes through reordering as above, the I/O scheduler could merge multiple FLUSHes into a single command.

4. On MD/RAID, BARRIER requires every backing device to quiesce before sending the low-level cache-flush, and all of those to finish before resuming each backing device. FLUSH doesn't require as much synchronising. (With per-file FLUSH; see 2; it could even avoid FLUSH altogether to some backing devices for small files).

In other words, FLUSH can be more relaxed than BARRIER inside the kernel. It's ironic that we think of fsync as stronger than fbarrier outside the kernel :-)

-- Jamie

Jamie Lokier

2010-05-04 20:17 UTC #22 lore raw

In reply to Jens Axboe

Quoted 9 lines · Jens Axboe

Quoted 2 lines · Rusty Russell
ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...
It would be nice to have a more fuller API for this, but the reality is that only the flush approach is really workable. Even just strict ordering of requests could only be supported on SCSI, and even there the kernel still lacks proper guarantees on error handling to prevent reordering there.

There's a few I/O scheduling differences that might be useful:

1. The I/O scheduler could freely move WRITEs before a FLUSH but not before a BARRIER. That might be useful for time-critical WRITEs, and those issued by high I/O priority.

3. By delaying FLUSHes through reordering as above, the I/O scheduler could merge multiple FLUSHes into a single command.

In other words, FLUSH can be more relaxed than BARRIER inside the kernel. It's ironic that we think of fsync as stronger than fbarrier outside the kernel :-)

-- Jamie

Rusty Russell

2010-05-05 04:58 UTC #23 lore raw

Quoted 11 lines · Jamie Lokier

Quoted 9 lines · Jens Axboe
Quoted 2 lines · Rusty Russell
ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...
It would be nice to have a more fuller API for this, but the reality is that only the flush approach is really workable. Even just strict ordering of requests could only be supported on SCSI, and even there the kernel still lacks proper guarantees on error handling to prevent reordering there.

Quoted 5 lines

There's a few I/O scheduling differences that might be useful:
1. The I/O scheduler could freely move WRITEs before a FLUSH but not before a BARRIER. That might be useful for time-critical WRITEs, and those issued by high I/O priority.

This is only because noone actually wants flushes or barriers, though I/O people seem to only offer that. We really want "<these writes> must occur before <this write>". That offers maximum choice to the I/O subsystem and potentially to smart (virtual?) disks.

Quoted 6 lines

2. The I/O scheduler could move WRITEs after a FLUSH if the FLUSH is only for data belonging to a particular file (e.g. fdatasync with no file size change, even on btrfs if O_DIRECT was used for the writes being committed). That would entail tagging FLUSHes and WRITEs with a fs-specific identifier (such as inode number), opaque to the scheduler which only checks equality.

This is closer. In userspace I'd be happy with a "all prior writes to this struct file before all future writes". Even if the original guarantees were stronger (ie. inode basis). We currently implement transactions using 4 fsync /msync pairs.

	write_recovery_data(fd);
	fsync(fd);
	msync(mmap);
	write_recovery_header(fd);
	fsync(fd);
	msync(mmap);
	overwrite_with_new_data(fd);
	fsync(fd);
	msync(mmap);
	remove_recovery_header(fd);
	fsync(fd);
	msync(mmap);

Yet we really only need ordering, not guarantees about it actually hitting disk before returning.

Quoted 3 lines

In other words, FLUSH can be more relaxed than BARRIER inside the kernel. It's ironic that we think of fsync as stronger than fbarrier outside the kernel :-)

It's an implementation detail; barrier has less flexibility because it has less information about what is required. I'm saying I want to give you as much information as I can, even if you don't use it yet.

Thanks,
Rusty.

Rusty Russell

2010-05-05 04:58 UTC #24 lore raw

In reply to Jamie Lokier

Quoted 11 lines · Jamie Lokier

Quoted 9 lines · Jens Axboe
Quoted 2 lines · Rusty Russell
ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...
It would be nice to have a more fuller API for this, but the reality is that only the flush approach is really workable. Even just strict ordering of requests could only be supported on SCSI, and even there the kernel still lacks proper guarantees on error handling to prevent reordering there.

Quoted 5 lines

There's a few I/O scheduling differences that might be useful:
1. The I/O scheduler could freely move WRITEs before a FLUSH but not before a BARRIER. That might be useful for time-critical WRITEs, and those issued by high I/O priority.

Quoted 6 lines

2. The I/O scheduler could move WRITEs after a FLUSH if the FLUSH is only for data belonging to a particular file (e.g. fdatasync with no file size change, even on btrfs if O_DIRECT was used for the writes being committed). That would entail tagging FLUSHes and WRITEs with a fs-specific identifier (such as inode number), opaque to the scheduler which only checks equality.

	write_recovery_data(fd);
	fsync(fd);
	msync(mmap);
	write_recovery_header(fd);
	fsync(fd);
	msync(mmap);
	overwrite_with_new_data(fd);
	fsync(fd);
	msync(mmap);
	remove_recovery_header(fd);
	fsync(fd);
	msync(mmap);

Yet we really only need ordering, not guarantees about it actually hitting disk before returning.

Quoted 3 lines

In other words, FLUSH can be more relaxed than BARRIER inside the kernel. It's ironic that we think of fsync as stronger than fbarrier outside the kernel :-)

Thanks,
Rusty.

Neil Brown

2010-05-05 06:03 UTC #25 lore raw

Quoted 35 lines · On Wed, 5 May 2010 14:28:41 +0930 Rusty Russell <[email protected]> wrote:

Quoted 11 lines · Jamie Lokier
Quoted 9 lines · Jens Axboe
Quoted 2 lines · Rusty Russell
ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...
It would be nice to have a more fuller API for this, but the reality is that only the flush approach is really workable. Even just strict ordering of requests could only be supported on SCSI, and even there the kernel still lacks proper guarantees on error handling to prevent reordering there.
Quoted 5 lines
There's a few I/O scheduling differences that might be useful:
1. The I/O scheduler could freely move WRITEs before a FLUSH but not before a BARRIER. That might be useful for time-critical WRITEs, and those issued by high I/O priority.
This is only because noone actually wants flushes or barriers, though I/O people seem to only offer that. We really want "<these writes> must occur before <this write>". That offers maximum choice to the I/O subsystem and potentially to smart (virtual?) disks.
Quoted 6 lines
2. The I/O scheduler could move WRITEs after a FLUSH if the FLUSH is only for data belonging to a particular file (e.g. fdatasync with no file size change, even on btrfs if O_DIRECT was used for the writes being committed). That would entail tagging FLUSHes and WRITEs with a fs-specific identifier (such as inode number), opaque to the scheduler which only checks equality.
This is closer. In userspace I'd be happy with a "all prior writes to this struct file before all future writes". Even if the original guarantees were stronger (ie. inode basis). We currently implement transactions using 4 fsync /msync pairs.

Quoted 12 lines

	write_recovery_data(fd);
	fsync(fd);
	msync(mmap);
	write_recovery_header(fd);
	fsync(fd);
	msync(mmap);
	overwrite_with_new_data(fd);
	fsync(fd);
	msync(mmap);
	remove_recovery_header(fd);
	fsync(fd);
	msync(mmap);

Seems over-zealous.
If the recovery_header held a strong checksum of the recovery_data you would not need the first fsync, and as long as you have two places to write recovery data, you don't need the 3rd and 4th syncs.
Just:
write_internally_checksummed_recovery_data_and_header_to_unused_log_space() fsync / msync
overwrite_with_new_data()

To recovery you choose the most recent log_space and replay the content. That may be a redundant operation, but that is no loss.

Also cannot see the point of msync if you have already performed an fsync, and if there is a point, I would expect you to call msync before fsync... Maybe there is some subtlety there that I am not aware of.

Quoted 11 lines

Yet we really only need ordering, not guarantees about it actually hitting disk before returning.
Quoted 3 lines
In other words, FLUSH can be more relaxed than BARRIER inside the kernel. It's ironic that we think of fsync as stronger than fbarrier outside the kernel :-)
It's an implementation detail; barrier has less flexibility because it has less information about what is required. I'm saying I want to give you as much information as I can, even if you don't use it yet.

Only we know that approach doesn't work.
People will learn that they don't need to give the extra information to still achieve the same result - just like they did with ext3 and fsync. Then when we improve the implementation to only provide the guarantees that you asked for, people will complain that they are getting empty files that they didn't expect.

The abstraction I would like to see is a simple 'barrier' that contains no data and has a filesystem-wide effect.

If a filesystem wanted a 'full' barrier such as the current BIO_RW_BARRER, it would send an empty barrier, then the data, then another empty barrier. (However I suspect most filesystems don't really need barriers on both sides.) A low level driver might merge these together if the underlying hardware supported that combined operation (which I believe some do).
I think this merging would be less complex that the current need to split a BIO_RW_BARRIER in to the three separate operations when only a flush is possible (I know it would make md code a lot nicer :-).

I would probably expose this to user-space as extra flags to sync_file_range:
SYNC_FILE_RANGE_BARRIER_BEFORE
SYNC_FILE_RANGE_BARRIER_AFTER

This would make it clear that a barrier does *not* imply a sync, it only applies to data for which a sync has already been requested. So data that has already been 'synced' is stored strictly before data which has not yet been submitted with write() (or by changing a mmapped area).
The barrier would still be filesystem wide in that if you SYNC_FILE_WRITE_WRITE one file, then SYNC_FILE_RANGE_BARRIER_BEFORE another file on the same filesystem, the pages scheduled in the first file would be affect by the barrier request on the second file.

Implementing this would probably require a new address_space_operation so that the filesystem would have a chance to ensure all necessary writes were queued before issuing the barrier.

NeilBrown

Neil Brown

2010-05-05 06:04 UTC #26 lore raw

Quoted 35 lines · On Wed, 5 May 2010 14:28:41 +0930 Rusty Russell <[email protected]> wrote:

Quoted 11 lines · Jamie Lokier
Quoted 9 lines · Jens Axboe
Quoted 2 lines · Rusty Russell
ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...
It would be nice to have a more fuller API for this, but the reality is that only the flush approach is really workable. Even just strict ordering of requests could only be supported on SCSI, and even there the kernel still lacks proper guarantees on error handling to prevent reordering there.
Quoted 5 lines
There's a few I/O scheduling differences that might be useful:
1. The I/O scheduler could freely move WRITEs before a FLUSH but not before a BARRIER. That might be useful for time-critical WRITEs, and those issued by high I/O priority.
This is only because noone actually wants flushes or barriers, though I/O people seem to only offer that. We really want "<these writes> must occur before <this write>". That offers maximum choice to the I/O subsystem and potentially to smart (virtual?) disks.
Quoted 6 lines
2. The I/O scheduler could move WRITEs after a FLUSH if the FLUSH is only for data belonging to a particular file (e.g. fdatasync with no file size change, even on btrfs if O_DIRECT was used for the writes being committed). That would entail tagging FLUSHes and WRITEs with a fs-specific identifier (such as inode number), opaque to the scheduler which only checks equality.
This is closer. In userspace I'd be happy with a "all prior writes to this struct file before all future writes". Even if the original guarantees were stronger (ie. inode basis). We currently implement transactions using 4 fsync /msync pairs.

Quoted 12 lines

	write_recovery_data(fd);
	fsync(fd);
	msync(mmap);
	write_recovery_header(fd);
	fsync(fd);
	msync(mmap);
	overwrite_with_new_data(fd);
	fsync(fd);
	msync(mmap);
	remove_recovery_header(fd);
	fsync(fd);
	msync(mmap);

To recovery you choose the most recent log_space and replay the content. That may be a redundant operation, but that is no loss.

Quoted 11 lines

Yet we really only need ordering, not guarantees about it actually hitting disk before returning.
Quoted 3 lines
In other words, FLUSH can be more relaxed than BARRIER inside the kernel. It's ironic that we think of fsync as stronger than fbarrier outside the kernel :-)
It's an implementation detail; barrier has less flexibility because it has less information about what is required. I'm saying I want to give you as much information as I can, even if you don't use it yet.

The abstraction I would like to see is a simple 'barrier' that contains no data and has a filesystem-wide effect.

I would probably expose this to user-space as extra flags to sync_file_range:
SYNC_FILE_RANGE_BARRIER_BEFORE
SYNC_FILE_RANGE_BARRIER_AFTER

Implementing this would probably require a new address_space_operation so that the filesystem would have a chance to ensure all necessary writes were queued before issuing the barrier.

NeilBrown

Rusty Russell

2010-05-06 06:05 UTC #27 lore raw

Quoted 60 lines · Neil Brown

Quoted 35 lines · On Wed, 5 May 2010 14:28:41 +0930 Rusty Russell <[email protected]> wrote:
Quoted 11 lines · Jamie Lokier
Quoted 9 lines · Jens Axboe
Quoted 2 lines · Rusty Russell
ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...
It would be nice to have a more fuller API for this, but the reality is that only the flush approach is really workable. Even just strict ordering of requests could only be supported on SCSI, and even there the kernel still lacks proper guarantees on error handling to prevent reordering there.
Quoted 5 lines
There's a few I/O scheduling differences that might be useful:
1. The I/O scheduler could freely move WRITEs before a FLUSH but not before a BARRIER. That might be useful for time-critical WRITEs, and those issued by high I/O priority.
This is only because noone actually wants flushes or barriers, though I/O people seem to only offer that. We really want "<these writes> must occur before <this write>". That offers maximum choice to the I/O subsystem and potentially to smart (virtual?) disks.
Quoted 6 lines
2. The I/O scheduler could move WRITEs after a FLUSH if the FLUSH is only for data belonging to a particular file (e.g. fdatasync with no file size change, even on btrfs if O_DIRECT was used for the writes being committed). That would entail tagging FLUSHes and WRITEs with a fs-specific identifier (such as inode number), opaque to the scheduler which only checks equality.
This is closer. In userspace I'd be happy with a "all prior writes to this struct file before all future writes". Even if the original guarantees were stronger (ie. inode basis). We currently implement transactions using 4 fsync /msync pairs.
Quoted 12 lines
	write_recovery_data(fd);
	fsync(fd);
	msync(mmap);
	write_recovery_header(fd);
	fsync(fd);
	msync(mmap);
	overwrite_with_new_data(fd);
	fsync(fd);
	msync(mmap);
	remove_recovery_header(fd);
	fsync(fd);
	msync(mmap);
Seems over-zealous.
If the recovery_header held a strong checksum of the recovery_data you would not need the first fsync, and as long as you have two places to write recovery data, you don't need the 3rd and 4th syncs.
Just:
write_internally_checksummed_recovery_data_and_header_to_unused_log_space() fsync / msync
overwrite_with_new_data()

Quoted 2 lines

To recovery you choose the most recent log_space and replay the content. That may be a redundant operation, but that is no loss.

I think you missed a checksum for the new data? Otherwise we can't tell if the new data is completely written. But yes, I will steal this scheme for TDB2, thanks!

In practice, it's the first sync which is glacial, the rest are pretty cheap.

Quoted 3 lines

Also cannot see the point of msync if you have already performed an fsync, and if there is a point, I would expect you to call msync before fsync... Maybe there is some subtlety there that I am not aware of.

I assume it's this from the msync man page:

msync() flushes changes made to the in-core copy of a file that was
mapped into memory using mmap(2) back to disk. Without use of this
call there is no guarantee that changes are written back before mun‐ map(2) is called.

Quoted 10 lines

Quoted 3 lines
It's an implementation detail; barrier has less flexibility because it has less information about what is required. I'm saying I want to give you as much information as I can, even if you don't use it yet.
Only we know that approach doesn't work.
People will learn that they don't need to give the extra information to still achieve the same result - just like they did with ext3 and fsync. Then when we improve the implementation to only provide the guarantees that you asked for, people will complain that they are getting empty files that they didn't expect.

I think that's an oversimplification: IIUC that occurred to people *not* using fsync(). They weren't using it because it was too slow. Providing a primitive which is as fast or faster and more specific doesn't have the same magnitude of social issues.

And we can't write userspace interfaces for idiots only.

Quoted 2 lines

The abstraction I would like to see is a simple 'barrier' that contains no data and has a filesystem-wide effect.

I think you lack ambition ;)

Thinking about the single-file use case (eg. kvm guest or tdb), isn't that suboptimal for md? Since you have to hand your barrier to every device whereas a file-wide primitive may theoretically only go to some.

Cheers,
Rusty.
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization

Rusty Russell

2010-05-06 06:05 UTC #28 lore raw

In reply to Neil Brown

Quoted 60 lines · Neil Brown

Quoted 35 lines · On Wed, 5 May 2010 14:28:41 +0930 Rusty Russell <[email protected]> wrote:
Quoted 11 lines · Jamie Lokier
Quoted 9 lines · Jens Axboe
Quoted 2 lines · Rusty Russell
ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...
It would be nice to have a more fuller API for this, but the reality is that only the flush approach is really workable. Even just strict ordering of requests could only be supported on SCSI, and even there the kernel still lacks proper guarantees on error handling to prevent reordering there.
Quoted 5 lines
There's a few I/O scheduling differences that might be useful:
1. The I/O scheduler could freely move WRITEs before a FLUSH but not before a BARRIER. That might be useful for time-critical WRITEs, and those issued by high I/O priority.
This is only because noone actually wants flushes or barriers, though I/O people seem to only offer that. We really want "<these writes> must occur before <this write>". That offers maximum choice to the I/O subsystem and potentially to smart (virtual?) disks.
Quoted 6 lines
2. The I/O scheduler could move WRITEs after a FLUSH if the FLUSH is only for data belonging to a particular file (e.g. fdatasync with no file size change, even on btrfs if O_DIRECT was used for the writes being committed). That would entail tagging FLUSHes and WRITEs with a fs-specific identifier (such as inode number), opaque to the scheduler which only checks equality.
This is closer. In userspace I'd be happy with a "all prior writes to this struct file before all future writes". Even if the original guarantees were stronger (ie. inode basis). We currently implement transactions using 4 fsync /msync pairs.
Quoted 12 lines
	write_recovery_data(fd);
	fsync(fd);
	msync(mmap);
	write_recovery_header(fd);
	fsync(fd);
	msync(mmap);
	overwrite_with_new_data(fd);
	fsync(fd);
	msync(mmap);
	remove_recovery_header(fd);
	fsync(fd);
	msync(mmap);
Seems over-zealous.
If the recovery_header held a strong checksum of the recovery_data you would not need the first fsync, and as long as you have two places to write recovery data, you don't need the 3rd and 4th syncs.
Just:
write_internally_checksummed_recovery_data_and_header_to_unused_log_space() fsync / msync
overwrite_with_new_data()

Quoted 2 lines

To recovery you choose the most recent log_space and replay the content. That may be a redundant operation, but that is no loss.

I think you missed a checksum for the new data? Otherwise we can't tell if the new data is completely written. But yes, I will steal this scheme for TDB2, thanks!

In practice, it's the first sync which is glacial, the rest are pretty cheap.

Quoted 3 lines

Also cannot see the point of msync if you have already performed an fsync, and if there is a point, I would expect you to call msync before fsync... Maybe there is some subtlety there that I am not aware of.

I assume it's this from the msync man page:

Quoted 10 lines

Quoted 3 lines
It's an implementation detail; barrier has less flexibility because it has less information about what is required. I'm saying I want to give you as much information as I can, even if you don't use it yet.
Only we know that approach doesn't work.
People will learn that they don't need to give the extra information to still achieve the same result - just like they did with ext3 and fsync. Then when we improve the implementation to only provide the guarantees that you asked for, people will complain that they are getting empty files that they didn't expect.

And we can't write userspace interfaces for idiots only.

Quoted 2 lines

The abstraction I would like to see is a simple 'barrier' that contains no data and has a filesystem-wide effect.

I think you lack ambition ;)

Cheers,
Rusty.

Jamie Lokier

2010-05-06 14:57 UTC #29 lore raw

Quoted 9 lines · Rusty Russell

Quoted 9 lines
Seems over-zealous.
If the recovery_header held a strong checksum of the recovery_data you would not need the first fsync, and as long as you have two places to write recovery data, you don't need the 3rd and 4th syncs.
Just:
write_internally_checksummed_recovery_data_and_header_to_unused_log_space() fsync / msync
overwrite_with_new_data()

Quoted 5 lines

Quoted 2 lines
To recovery you choose the most recent log_space and replay the content. That may be a redundant operation, but that is no loss.
I think you missed a checksum for the new data? Otherwise we can't tell if the new data is completely written.

The data checksum can go in the recovery-data block. If there's enough slack in the log, by the time that recovery-data block is overwritten, you can be sure that an fsync has been done for that data (by a later commit).

Quoted 1 line

But yes, I will steal this scheme for TDB2, thanks!

Take a look at the filesystems. I think ext4 did some optimisations in this area, and that checksums had to be added anyway due to a subtle replay-corruption problem that happens when the log is partially corrupted, and followed by non-corrupt blocks.

Also, you can remove even more fsyncs by adding a bit of slack to the data space and writing into unused/fresh areas some of the time - i.e. a bit like btrfs/zfs or anything log-structured, but you don't have to go all the way with that.

Quoted 1 line

In practice, it's the first sync which is glacial, the rest are pretty cheap.

The 3rd and 4th fsyncs imply a disk seek each, just because the preceding writes are to different areas of the disk. Seeks are quite slow - but not as slow as ext3 fsyncs :-) What do you mean by cheap? That it's only a couple of seeks, or that you don't see even that?

Quoted 11 lines

Quoted 4 lines
Also cannot see the point of msync if you have already performed an fsync, and if there is a point, I would expect you to call msync before fsync... Maybe there is some subtlety there that I am not aware of.
I assume it's this from the msync man page:
msync() flushes changes made to the in-core copy of a file that was
mapped into memory using mmap(2) back to disk. Without use of this
call there is no guarantee that changes are written back before mun‐ map(2) is called.

Historically, that means msync() ensures dirty mapping data is written to the file as if with write(), and that mapping pages are removed or refreshed to get the effect of read() (possibly a lazy one). It's more obvious in the early mmap implementations where mappings don't share pages with the filesystem cache, so msync() has explicit behaviour.

Like with write(), after calling msync() you would then call fsync() to ensure the data is flushed to disk.

If you've been calling fsync then msync, I guess that's another fine example of how these function are so hard to test, that they aren't.

Historically on Linux, msync has been iffy on some architectures, and I'm still not sure it has the same semantics as other unixes. fsync as we know has also been iffy, and even now that fsync is tidier it does not always issue a hardware-level cache commit.

But then historically writable mmap has been iffy on a boatload of unixes.

Quoted 11 lines

Quoted 10 lines
Quoted 3 lines
It's an implementation detail; barrier has less flexibility because it has less information about what is required. I'm saying I want to give you as much information as I can, even if you don't use it yet.
Only we know that approach doesn't work.
People will learn that they don't need to give the extra information to still achieve the same result - just like they did with ext3 and fsync. Then when we improve the implementation to only provide the guarantees that you asked for, people will complain that they are getting empty files that they didn't expect.

Quoted 4 lines

I think that's an oversimplification: IIUC that occurred to people *not* using fsync(). They weren't using it because it was too slow. Providing a primitive which is as fast or faster and more specific doesn't have the same magnitude of social issues.

I agree with Rusty. Let's make it perform well so there is no reason to deliberately avoid using it, and let's make say what apps actually want to request without being way too strong.

And please, if anyone has ideas on how we could make correct use of these functions *testable* by app authors, I'm all ears. Right now it is quite difficult - pulling power on hard disks mid-transaction is not a convenient method :)

Quoted 8 lines

Quoted 2 lines
The abstraction I would like to see is a simple 'barrier' that contains no data and has a filesystem-wide effect.
I think you lack ambition ;)
Thinking about the single-file use case (eg. kvm guest or tdb), isn't that suboptimal for md? Since you have to hand your barrier to every device whereas a file-wide primitive may theoretically only go to some.

Yes.

Note that database-like programs still need fsync-like behaviour *sometimes*: The "D" in ACID depends on it, and the "C" in ACID also depends on it where multiple files are involved which must contain consistent data with each other after crash/recovery (Perhaps Samba depends on this?)

Single-file sync is valuable just like single-file barrier, and so is the combination.

Since you mentioned ambition, think about multi-file updates. They're analogous in userspace to MD's barrier/sync requirements in kernelspace.

One API that supports multi-file update barriers is "long aio-fsync":
Something which returns when the data in earlier writes (to one file) is committed, but does not force the commit to happen more quickly than normal. Both single-file barriers (like you want for TDB) and multi-file barriers can be implemented on top of that, but it's much more difficult to use than an fbarrier() syscall, which is only suitable for single-file. But I wonder if there would be many users of fbarrier() who aren't perfectly capable of using something else if needed.

-- Jamie
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/virtualization

Jamie Lokier

2010-05-06 14:57 UTC #30 lore raw

Quoted 9 lines · Rusty Russell

Quoted 9 lines
Seems over-zealous.
If the recovery_header held a strong checksum of the recovery_data you would not need the first fsync, and as long as you have two places to write recovery data, you don't need the 3rd and 4th syncs.
Just:
write_internally_checksummed_recovery_data_and_header_to_unused_log_space() fsync / msync
overwrite_with_new_data()

Quoted 5 lines

Quoted 2 lines
To recovery you choose the most recent log_space and replay the content. That may be a redundant operation, but that is no loss.
I think you missed a checksum for the new data? Otherwise we can't tell if the new data is completely written.

Quoted 1 line

But yes, I will steal this scheme for TDB2, thanks!

Quoted 1 line

In practice, it's the first sync which is glacial, the rest are pretty cheap.

Quoted 11 lines

Quoted 4 lines
Also cannot see the point of msync if you have already performed an fsync, and if there is a point, I would expect you to call msync before fsync... Maybe there is some subtlety there that I am not aware of.
I assume it's this from the msync man page:
msync() flushes changes made to the in-core copy of a file that was
mapped into memory using mmap(2) back to disk. Without use of this
call there is no guarantee that changes are written back before mun‐ map(2) is called.

Like with write(), after calling msync() you would then call fsync() to ensure the data is flushed to disk.

If you've been calling fsync then msync, I guess that's another fine example of how these function are so hard to test, that they aren't.

But then historically writable mmap has been iffy on a boatload of unixes.

Quoted 11 lines

Quoted 10 lines
Quoted 3 lines
It's an implementation detail; barrier has less flexibility because it has less information about what is required. I'm saying I want to give you as much information as I can, even if you don't use it yet.
Only we know that approach doesn't work.
People will learn that they don't need to give the extra information to still achieve the same result - just like they did with ext3 and fsync. Then when we improve the implementation to only provide the guarantees that you asked for, people will complain that they are getting empty files that they didn't expect.

Quoted 4 lines

I think that's an oversimplification: IIUC that occurred to people *not* using fsync(). They weren't using it because it was too slow. Providing a primitive which is as fast or faster and more specific doesn't have the same magnitude of social issues.

I agree with Rusty. Let's make it perform well so there is no reason to deliberately avoid using it, and let's make say what apps actually want to request without being way too strong.

Quoted 8 lines

Quoted 2 lines
The abstraction I would like to see is a simple 'barrier' that contains no data and has a filesystem-wide effect.
I think you lack ambition ;)
Thinking about the single-file use case (eg. kvm guest or tdb), isn't that suboptimal for md? Since you have to hand your barrier to every device whereas a file-wide primitive may theoretically only go to some.

Yes.

Single-file sync is valuable just like single-file barrier, and so is the combination.

Since you mentioned ambition, think about multi-file updates. They're analogous in userspace to MD's barrier/sync requirements in kernelspace.

-- Jamie

Jamie Lokier

2010-05-06 15:25 UTC #31 lore raw

Quoted 14 lines · Rusty Russell

Quoted 13 lines · Jamie Lokier
Quoted 9 lines · Jens Axboe
Quoted 2 lines · Rusty Russell
ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...
It would be nice to have a more fuller API for this, but the reality is that only the flush approach is really workable. Even just strict ordering of requests could only be supported on SCSI, and even there the kernel still lacks proper guarantees on error handling to prevent reordering there.
There's a few I/O scheduling differences that might be useful:

Quoted 8 lines

Quoted 3 lines
1. The I/O scheduler could freely move WRITEs before a FLUSH but not before a BARRIER. That might be useful for time-critical WRITEs, and those issued by high I/O priority.
This is only because noone actually wants flushes or barriers, though I/O people seem to only offer that. We really want "<these writes> must occur before <this write>". That offers maximum choice to the I/O subsystem and potentially to smart (virtual?) disks.

We do want flushes for the "D" in ACID - such things as after receiving a mail, or blog update into a database file (could be TDB), and confirming that to the sender, to have high confidence that the update won't disappear on system crash or power failure.

Less obviously, it's also needed for the "C" in ACID when more than one file is involved. "C" is about differently updated things staying consistent with each other.

For example, imagine you have a TDB file mapping Samba usernames to passwords, and another mapping Samba usernames to local usernames. (I don't know if you do this; it's just an illustration).

To rename a Samba user involves updating both. Let's ignore transient transactional issues :-) and just think about what happens with per-file barriers and no sync, when a crash happens long after the updates, and before the system has written out all data and issued low level cache flushes.

After restarting, due to lack of sync, the Samba username could be present in one file and not the other.

Quoted 28 lines

Quoted 6 lines
2. The I/O scheduler could move WRITEs after a FLUSH if the FLUSH is only for data belonging to a particular file (e.g. fdatasync with no file size change, even on btrfs if O_DIRECT was used for the writes being committed). That would entail tagging FLUSHes and WRITEs with a fs-specific identifier (such as inode number), opaque to the scheduler which only checks equality.
This is closer. In userspace I'd be happy with a "all prior writes to this struct file before all future writes". Even if the original guarantees were stronger (ie. inode basis). We currently implement transactions using 4 fsync /msync pairs.
	write_recovery_data(fd);
	fsync(fd);
	msync(mmap);
	write_recovery_header(fd);
	fsync(fd);
	msync(mmap);
	overwrite_with_new_data(fd);
	fsync(fd);
	msync(mmap);
	remove_recovery_header(fd);
	fsync(fd);
	msync(mmap);
Yet we really only need ordering, not guarantees about it actually hitting disk before returning.

Quoted 7 lines

Quoted 3 lines
In other words, FLUSH can be more relaxed than BARRIER inside the kernel. It's ironic that we think of fsync as stronger than fbarrier outside the kernel :-)
It's an implementation detail; barrier has less flexibility because it has less information about what is required. I'm saying I want to give you as much information as I can, even if you don't use it yet.

I agree, and I've started a few threads about it over the last couple of years.

An fsync_range() system call would be very easy to use and (most importantly) easy to understand.

With optional flags to weaken it (into fdatasync, barrier without sync, sync without barrier, one-sided barrier, no lowlevel cache-flush, don't rush, etc.), it would be very versatile, and still easy to understand.

With an AIO version, and another flag meaning don't rush, just return when satisfied, and I suspect it would be useful for the most demanding I/O apps.

-- Jamie

Jamie Lokier

2010-05-06 15:25 UTC #32 lore raw

Quoted 14 lines · Rusty Russell

Quoted 13 lines · Jamie Lokier
Quoted 9 lines · Jens Axboe
Quoted 2 lines · Rusty Russell
ISTR someone mentioning a desire for such an API years ago, so CC'ing the usual I/O suspects...
It would be nice to have a more fuller API for this, but the reality is that only the flush approach is really workable. Even just strict ordering of requests could only be supported on SCSI, and even there the kernel still lacks proper guarantees on error handling to prevent reordering there.
There's a few I/O scheduling differences that might be useful:

Quoted 8 lines

Quoted 3 lines
1. The I/O scheduler could freely move WRITEs before a FLUSH but not before a BARRIER. That might be useful for time-critical WRITEs, and those issued by high I/O priority.
This is only because noone actually wants flushes or barriers, though I/O people seem to only offer that. We really want "<these writes> must occur before <this write>". That offers maximum choice to the I/O subsystem and potentially to smart (virtual?) disks.

Less obviously, it's also needed for the "C" in ACID when more than one file is involved. "C" is about differently updated things staying consistent with each other.

For example, imagine you have a TDB file mapping Samba usernames to passwords, and another mapping Samba usernames to local usernames. (I don't know if you do this; it's just an illustration).

After restarting, due to lack of sync, the Samba username could be present in one file and not the other.

Quoted 28 lines

Quoted 6 lines
2. The I/O scheduler could move WRITEs after a FLUSH if the FLUSH is only for data belonging to a particular file (e.g. fdatasync with no file size change, even on btrfs if O_DIRECT was used for the writes being committed). That would entail tagging FLUSHes and WRITEs with a fs-specific identifier (such as inode number), opaque to the scheduler which only checks equality.
This is closer. In userspace I'd be happy with a "all prior writes to this struct file before all future writes". Even if the original guarantees were stronger (ie. inode basis). We currently implement transactions using 4 fsync /msync pairs.
	write_recovery_data(fd);
	fsync(fd);
	msync(mmap);
	write_recovery_header(fd);
	fsync(fd);
	msync(mmap);
	overwrite_with_new_data(fd);
	fsync(fd);
	msync(mmap);
	remove_recovery_header(fd);
	fsync(fd);
	msync(mmap);
Yet we really only need ordering, not guarantees about it actually hitting disk before returning.

Quoted 7 lines

Quoted 3 lines
In other words, FLUSH can be more relaxed than BARRIER inside the kernel. It's ironic that we think of fsync as stronger than fbarrier outside the kernel :-)
It's an implementation detail; barrier has less flexibility because it has less information about what is required. I'm saying I want to give you as much information as I can, even if you don't use it yet.

I agree, and I've started a few threads about it over the last couple of years.

An fsync_range() system call would be very easy to use and (most importantly) easy to understand.

With an AIO version, and another flag meaning don't rush, just return when satisfied, and I suspect it would be useful for the most demanding I/O apps.

-- Jamie

Christoph Hellwig

2010-05-04 10:05 UTC #33 lore raw

Quoted 9 lines · Rusty Russell

Quoted 5 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.
ISTR Christoph had withdrawn some patches in this area, and was waiting for him to resubmit?

Any patches I've withdrawn in this area are withdrawn for good. But what I really need to do is to review Michaels spec updates, sorry.

UI'll get back to it today.

Christoph Hellwig

2010-05-04 10:06 UTC #34 lore raw

Quoted 9 lines · Rusty Russell

Quoted 5 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.
ISTR Christoph had withdrawn some patches in this area, and was waiting for him to resubmit?

Any patches I've withdrawn in this area are withdrawn for good. But what I really need to do is to review Michaels spec updates, sorry.

UI'll get back to it today.

Jamie Lokier

2010-05-04 20:32 UTC #35 lore raw

Quoted 14 lines · Rusty Russell

Quoted 5 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.
ISTR Christoph had withdrawn some patches in this area, and was waiting for him to resubmit?
I've given up on figuring out the block device. What seem to me to be sane semantics along the lines of memory barriers are foreign to disk people: they want (and depend on) flushing everywhere.

Quoted 5 lines

For example, tdb transactions do not require a flush, they only require what I would call a barrier: that prior data be written out before any future data. Surely that would be more efficient in general than a flush! In fact, TDB wants only writes to *that file* (and metadata) written out first; it has no ordering issues with other I/O on the same device.

I've just posted elsewhere on this thread, that an I/O level flush can be more efficient than an I/O level barrier (implemented using a cache-flush really), because the barrier has stricter ordering requirements at the I/O scheduling level.

By the time you work up to tdb, another way to think of it is distinguishing "eager fsync" from "fsync but I'm not in a hurry - delay as long as is convenient". The latter makes much more sense with AIO.

Quoted 3 lines

A generic I/O interface would allow you to specify "this request depends on these outstanding requests" and leave it at that. It might have some sync flush command for dumb applications and OSes.

For filesystems, it would probably be easy to label in-place overwrites and fdatasync data flushes when there's no file extension with an opqaue per-file identifier for certain operations. Typically over-writing in place and fdatasync would match up and wouldn't need ordering against anything else. Other operations would tend to get labelled as ordered against everything including these.

-- Jamie

Jamie Lokier

2010-05-04 20:33 UTC #36 lore raw

Quoted 14 lines · Rusty Russell

Quoted 5 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.
ISTR Christoph had withdrawn some patches in this area, and was waiting for him to resubmit?
I've given up on figuring out the block device. What seem to me to be sane semantics along the lines of memory barriers are foreign to disk people: they want (and depend on) flushing everywhere.

Quoted 5 lines

For example, tdb transactions do not require a flush, they only require what I would call a barrier: that prior data be written out before any future data. Surely that would be more efficient in general than a flush! In fact, TDB wants only writes to *that file* (and metadata) written out first; it has no ordering issues with other I/O on the same device.

Quoted 3 lines

A generic I/O interface would allow you to specify "this request depends on these outstanding requests" and leave it at that. It might have some sync flush command for dumb applications and OSes.

-- Jamie

Christoph Hellwig

2010-05-04 18:54 UTC #37 lore raw

Quoted 5 lines · Michael S. Tsirkin

I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.

The whole virtio-blk interface is full of warts. It has been extended rather ad-hoc, so that is rather expected.

Quoted 4 lines

One issue I struggled with especially is how type
field mixes bits and non-bit values. I ended up
simply defining all legal values, so that we have
CMD = 2, CMD_OUT = 3 and so on.

It's basically a complete mess without much logic behind it.

Quoted 4 lines

+\change_unchanged
virtio-spec.lyx +3 -0
+the high bit
+\change_inserted 0 1266497301
+ (VIRTIO_BLK_T_BARRIER)

virtio-spec.lyx struct virtio_blk_req { · line 4979

Quoted 10 lines

virtio-spec.lyx +10 -0

+\change_unchanged
+ indicates that this request acts as a barrier and that all preceeding requests
+ must be complete before this one, and all following requests must not be
+ started until this is complete.
+
+\change_inserted 0 1266504385
+ Note that a barrier does not flush caches in the underlying backend device
+ in host, and thus does not serve as data consistency guarantee.
+ Driver must use FLUSH request to flush the host cache.
+\change_unchanged

I'm not sure it's even worth documenting it. I can't see any way to actually implement safe behaviour with the VIRTIO_BLK_T_BARRIER-style barriers.

Btw, did I mention that .lyx is a a really horrible format to review diffs for? Plain latex would be a lot better..

Michael S. Tsirkin

author maintainer

2010-05-04 18:56 UTC #38 lore raw

Quoted 27 lines · Christoph Hellwig

Quoted 5 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.
The whole virtio-blk interface is full of warts. It has been extended rather ad-hoc, so that is rather expected.
Quoted 4 lines
One issue I struggled with especially is how type
field mixes bits and non-bit values. I ended up
simply defining all legal values, so that we have
CMD = 2, CMD_OUT = 3 and so on.
It's basically a complete mess without much logic behind it.
Quoted 10 lines
+\change_unchanged
virtio-spec.lyx +9 -0
+the high bit
+\change_inserted 0 1266497301
+ (VIRTIO_BLK_T_BARRIER)
+\change_unchanged
+ indicates that this request acts as a barrier and that all preceeding requests
+ must be complete before this one, and all following requests must not be
+ started until this is complete.
+
+\change_inserted 0 1266504385

Quoted 8 lines

Quoted 4 lines
+ Note that a barrier does not flush caches in the underlying backend device
+ in host, and thus does not serve as data consistency guarantee.
+ Driver must use FLUSH request to flush the host cache. +\change_unchanged
I'm not sure it's even worth documenting it. I can't see any way to actually implement safe behaviour with the VIRTIO_BLK_T_BARRIER-style barriers.

lguest seems to still use this.
I guess if you have a reliable host, VIRTIO_BLK_T_BARRIER is enough?

Quoted 2 lines

Btw, did I mention that .lyx is a a really horrible format to review diffs for? Plain latex would be a lot better..

Michael S. Tsirkin

author maintainer

2010-05-04 19:00 UTC #39 lore raw

Quoted 27 lines · Christoph Hellwig

Quoted 5 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.
The whole virtio-blk interface is full of warts. It has been extended rather ad-hoc, so that is rather expected.
Quoted 4 lines
One issue I struggled with especially is how type
field mixes bits and non-bit values. I ended up
simply defining all legal values, so that we have
CMD = 2, CMD_OUT = 3 and so on.
It's basically a complete mess without much logic behind it.
Quoted 10 lines
+\change_unchanged
virtio-spec.lyx +9 -0
+the high bit
+\change_inserted 0 1266497301
+ (VIRTIO_BLK_T_BARRIER)
+\change_unchanged
+ indicates that this request acts as a barrier and that all preceeding requests
+ must be complete before this one, and all following requests must not be
+ started until this is complete.
+
+\change_inserted 0 1266504385

Quoted 8 lines

Quoted 4 lines
+ Note that a barrier does not flush caches in the underlying backend device
+ in host, and thus does not serve as data consistency guarantee.
+ Driver must use FLUSH request to flush the host cache. +\change_unchanged
I'm not sure it's even worth documenting it. I can't see any way to actually implement safe behaviour with the VIRTIO_BLK_T_BARRIER-style barriers.

lguest seems to still use this.
I guess if you have a reliable host, VIRTIO_BLK_T_BARRIER is enough?

Quoted 2 lines

Btw, did I mention that .lyx is a a really horrible format to review diffs for? Plain latex would be a lot better..

Michael S. Tsirkin

author maintainer

2010-05-04 18:58 UTC #40 lore raw

Quoted 37 lines · Michael S. Tsirkin

Quoted 27 lines · Christoph Hellwig
Quoted 5 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.
The whole virtio-blk interface is full of warts. It has been extended rather ad-hoc, so that is rather expected.
Quoted 4 lines
One issue I struggled with especially is how type
field mixes bits and non-bit values. I ended up
simply defining all legal values, so that we have
CMD = 2, CMD_OUT = 3 and so on.
It's basically a complete mess without much logic behind it.
Quoted 10 lines
+\change_unchanged
virtio-spec.lyx +9 -0
+the high bit
+\change_inserted 0 1266497301
+ (VIRTIO_BLK_T_BARRIER)
+\change_unchanged
+ indicates that this request acts as a barrier and that all preceeding requests
+ must be complete before this one, and all following requests must not be
+ started until this is complete.
+
+\change_inserted 0 1266504385
Quoted 8 lines
Quoted 4 lines
+ Note that a barrier does not flush caches in the underlying backend device
+ in host, and thus does not serve as data consistency guarantee.
+ Driver must use FLUSH request to flush the host cache. +\change_unchanged
I'm not sure it's even worth documenting it. I can't see any way to actually implement safe behaviour with the VIRTIO_BLK_T_BARRIER-style barriers.

Quoted 1 line

lguest seems to still use this.

Sorry, it doesn't. No idea why I thought it does.

Quoted 4 lines

I guess if you have a reliable host, VIRTIO_BLK_T_BARRIER is enough?
Quoted 2 lines
Btw, did I mention that .lyx is a a really horrible format to review diffs for? Plain latex would be a lot better..

Michael S. Tsirkin

author maintainer

2010-05-04 19:02 UTC #41 lore raw

Quoted 37 lines · Michael S. Tsirkin

Quoted 27 lines · Christoph Hellwig
Quoted 5 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.
The whole virtio-blk interface is full of warts. It has been extended rather ad-hoc, so that is rather expected.
Quoted 4 lines
One issue I struggled with especially is how type
field mixes bits and non-bit values. I ended up
simply defining all legal values, so that we have
CMD = 2, CMD_OUT = 3 and so on.
It's basically a complete mess without much logic behind it.
Quoted 10 lines
+\change_unchanged
virtio-spec.lyx +9 -0
+the high bit
+\change_inserted 0 1266497301
+ (VIRTIO_BLK_T_BARRIER)
+\change_unchanged
+ indicates that this request acts as a barrier and that all preceeding requests
+ must be complete before this one, and all following requests must not be
+ started until this is complete.
+
+\change_inserted 0 1266504385
Quoted 8 lines
Quoted 4 lines
+ Note that a barrier does not flush caches in the underlying backend device
+ in host, and thus does not serve as data consistency guarantee.
+ Driver must use FLUSH request to flush the host cache. +\change_unchanged
I'm not sure it's even worth documenting it. I can't see any way to actually implement safe behaviour with the VIRTIO_BLK_T_BARRIER-style barriers.

Quoted 1 line

lguest seems to still use this.

Sorry, it doesn't. No idea why I thought it does.

Quoted 4 lines

I guess if you have a reliable host, VIRTIO_BLK_T_BARRIER is enough?
Quoted 2 lines
Btw, did I mention that .lyx is a a really horrible format to review diffs for? Plain latex would be a lot better..

Rusty Russell

2010-05-05 05:00 UTC #42 lore raw

Quoted 37 lines · Christoph Hellwig

Quoted 5 lines · Michael S. Tsirkin
I took a stub at documenting CMD and FLUSH request types in virtio block. Christoph, could you look over this please?
I note that the interface seems full of warts to me, this might be a first step to cleaning them.
The whole virtio-blk interface is full of warts. It has been extended rather ad-hoc, so that is rather expected.
Quoted 4 lines
One issue I struggled with especially is how type
field mixes bits and non-bit values. I ended up
simply defining all legal values, so that we have
CMD = 2, CMD_OUT = 3 and so on.
It's basically a complete mess without much logic behind it.
Quoted 4 lines
+\change_unchanged
virtio-spec.lyx +3 -0
+the high bit
+\change_inserted 0 1266497301
+ (VIRTIO_BLK_T_BARRIER)
virtio-spec.lyx struct virtio_blk_req { · line 4979
Quoted 10 lines
virtio-spec.lyx +10 -0
+\change_unchanged
+ indicates that this request acts as a barrier and that all preceeding requests
+ must be complete before this one, and all following requests must not be
+ started until this is complete.
+
+\change_inserted 0 1266504385
+ Note that a barrier does not flush caches in the underlying backend device
+ in host, and thus does not serve as data consistency guarantee.
+ Driver must use FLUSH request to flush the host cache.
+\change_unchanged
I'm not sure it's even worth documenting it. I can't see any way to actually implement safe behaviour with the VIRTIO_BLK_T_BARRIER-style barriers.

Quoted 2 lines

Btw, did I mention that .lyx is a a really horrible format to review diffs for? Plain latex would be a lot better..

Yeah, or export as text post that diff for content review. I do like the patches to the lyx source though (I check all versions into revision control, before and after merging changes, which makes it easy to produce annotated versions).

Cheers,
Rusty.

Quoted 1 line

Rusty Russell

2010-05-05 05:00 UTC #43 lore raw