Support streaming to an intermediate layer

Hi,

thank you Kevin and Eric for your comments. I addressed them all in this series.

Berto

v13:
- Patch 10: explain that the 'base-node' parameter can be used to detect whether we support streaming to an intermediate node.
- Patch 12: Revert the blkdebug removal, it is indeed necessary to keep it.
- Patch 15: Fix the block-commit call in test_stream_commit().
- Patch 19: Add a new test to check that 'base' and 'base-node' cannot be specified at the same time.

v12: https://lists.gnu.org/archive/html/qemu-block/2016-10/msg01085.html
- Rebase on top of the current master.
- Patch 1: Add a note explaining that no new block jobs or BDSs can be added between bdrv_drain_all_begin/end().

v11: https://lists.gnu.org/archive/html/qemu-block/2016-10/msg00518.html
- Patches 1-2: Add bdrv_drain_all_{begin,end}() and use these functions in bdrv_reopen_multiple() instead of pausing all block jobs.
- Patch 3: Don't unblock BLOCK_OP_TYPE_DATAPLANE in block_job_add_bdrv()
- Patch 7: In commit_start(), don't call block_job_add_bdrv() twice on the active node.
- Patch 10: In qmp_block_stream(), don't check twice for op blockers in 'bs'.
- Patch 11: Clarify that live-block-ops.txt is incomplete and it only covers the block-stream operation.
- Patches 18-19: Add a 'base-node' parameter to 'block-stream', and iotests for it.

v10: https://lists.gnu.org/archive/html/qemu-devel/2016-10/msg01047.html
- Rebase the code.
- Pause block jobs during bdrv_reopen().
- Allow jobs in parallel again, and modify jobs to block all involved nodes, not just the root one.
- Add more tests.

v9: https://lists.gnu.org/archive/html/qemu-devel/2016-04/msg00411.html
- Rebase the code
- Block the active layer in order to forbid other block jobs in the same chain.
- Split the patch that adds block_job_next() into 4 (new patches 1-4).
- Replace the test that performs several block-stream operations in parallel with one that check that they're forbidden.
- Remove patches that have already been merged.

v8: https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg05754.html
- Rebased on top of Stefan's block branch (0a35bce416)
- The loop that pauses the block jobs in bdrv_drain_all() is now split in two: one that iterates the list of block jobs to stop them, and one that iterates the root bds in order to get the aio contexts.

v7: https://lists.gnu.org/archive/html/qemu-devel/2015-05/msg02580.html
- Rebased against the current master
- Updated bdrv_drain_all() to use the new block_job_next() API.

v6: https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg03046.html
- fix the no-op test following Max's suggestions

v5: https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg03006.html
- Fix a few typos
- Minor documentation updates
- Update test_stream_partial() to test no-ops
- New test case: test_stream_parallel()
- New test case: test_stream_overlapping()

v4: https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg01878.html
- Refactor find_block_job to use the error from bdrv_lookup_bs()
- Don't use QERR_DEVICE_IN_USE in block_job_create() since we can be dealing with nodes now.
- Fix @device comment in the BlockJobInfo documentation
- stream_start(): simplify the bdrv_reopen() call and use bdrv_get_device_or_node_name() for error messages.
- Use a different variable name for BlockDriverState *i
- Documentation fixes in docs/live-block-ops.txt
- Update iotest 30 since now test_device_not_found() returns GenericError
- Fix test case test_stream_partial()
- Add new test case test_stream_intermediate()
- Fix typos

v3: https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg00806.html
- Keep a list of block jobs and make qmp_query_block_jobs() iterate over it.

v2: https://lists.gnu.org/archive/html/qemu-devel/2015-03/msg04798.html
- The 'block-stream' command does not have a 'node-name' parameter anymore and reuses 'device' for that purpose.
- Block jobs can now be owned by any intermediate node, and not just by the ones at the root. query-block-jobs is updated to reflect that change.
- The 'device' parameter of all 'block-job-*' commands can now take a node name.
- The BlockJobInfo type and all BLOCK_JOB_* events report the node name in the 'device' field if the node does not have a device name.
- All intermediate nodes are blocked (and checked for blockers) during the streaming operation.

v1: https://lists.gnu.org/archive/html/qemu-devel/2015-02/msg04116.html

backport-diff against v12:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/19:[----] [--] 'block: Add bdrv_drain_all_{begin,end}()' 002/19:[----] [--] 'block: Pause all jobs during bdrv_reopen_multiple()' 003/19:[----] [--] 'block: Add block_job_add_bdrv()'
004/19:[----] [--] 'block: Use block_job_add_bdrv() in mirror_start_job()' 005/19:[----] [--] 'block: Use block_job_add_bdrv() in backup_start()' 006/19:[----] [--] 'block: Check blockers in all nodes involved in a block-commit job' 007/19:[----] [--] 'block: Block all nodes involved in the block-commit operation' 008/19:[----] [--] 'block: Block all intermediate nodes in commit_active_start()' 009/19:[----] [--] 'block: Support streaming to an intermediate layer' 010/19:[0005] [FC] 'block: Add QMP support for streaming to an intermediate layer' 011/19:[----] [--] 'docs: Document how to stream to an intermediate layer' 012/19:[0002] [FC] 'qemu-iotests: Test streaming to an intermediate layer' 013/19:[----] [--] 'qemu-iotests: Test block-stream operations in parallel' 014/19:[----] [--] 'qemu-iotests: Test overlapping stream and commit operations' 015/19:[0002] [FC] 'qemu-iotests: Test block-stream and block-commit in parallel' 016/19:[----] [--] 'qemu-iotests: Add iotests.supports_quorum()' 017/19:[----] [--] 'qemu-iotests: Test streaming to a Quorum child' 018/19:[----] [--] 'block: Add 'base-node' parameter to the 'block-stream' command' 019/19:[0004] [FC] 'qemu-iotests: Test the 'base-node' parameter of 'block-stream''

Alberto Garcia (19):
block: Add bdrv_drain_all_{begin,end}()
block: Pause all jobs during bdrv_reopen_multiple()
block: Add block_job_add_bdrv()
block: Use block_job_add_bdrv() in mirror_start_job()
block: Use block_job_add_bdrv() in backup_start()
block: Check blockers in all nodes involved in a block-commit job
block: Block all nodes involved in the block-commit operation
block: Block all intermediate nodes in commit_active_start()
block: Support streaming to an intermediate layer
block: Add QMP support for streaming to an intermediate layer
docs: Document how to stream to an intermediate layer
qemu-iotests: Test streaming to an intermediate layer
qemu-iotests: Test block-stream operations in parallel
qemu-iotests: Test overlapping stream and commit operations
qemu-iotests: Test block-stream and block-commit in parallel
qemu-iotests: Add iotests.supports_quorum()
qemu-iotests: Test streaming to a Quorum child
block: Add 'base-node' parameter to the 'block-stream' command
qemu-iotests: Test the 'base-node' parameter of 'block-stream'

 block.c                       |   9 +-
 block/backup.c                |   5 +-
 block/commit.c                |  14 ++
 block/io.c                    |  27 +++-
 block/mirror.c                |  11 +-
 block/stream.c                |  24 ++++
 blockdev.c                    |  41 +++++-
 blockjob.c                    |  17 ++-
 docs/live-block-ops.txt       |  36 +++--
 docs/qmp-commands.txt         |   7 +-
 hmp.c                         |   2 +-
 include/block/block.h         |   2 +
 include/block/blockjob.h      |  14 ++
 qapi/block-core.json          |  21 ++-
 tests/qemu-iotests/030        | 313 +++++++++++++++++++++++++++++++++++++++++-
 tests/qemu-iotests/030.out    |   4 +-
 tests/qemu-iotests/041        |  27 ++--
 tests/qemu-iotests/139        |   3 +-
 tests/qemu-iotests/iotests.py |   5 +-
 19 files changed, 525 insertions(+), 57 deletions(-)

2.9.3