qemu-io: add -V flag for read sub-command
PATCH 1/1 qemu-io: add -V flag for read sub-command
The problem this patch is trying to address is libguestfs behavior on the appliance startup. It starts supporting to use root=UUID definition in the kernel command line of its root filesystem using
file -- /usr/lib64/guestfs/appliance/root
This works fine with RAW image, but we are using QCOW2 as a storage to save a bit of file space and in this case we get
QEMU QCOW Image (v3), 1610612736 bytes
instead of UUID of the root filesystem.
The solution is very simple - we should dump first 256k of the image file like the follows
qemu-io -c "read -V 0 256k" appliance | file -
which will provide correct result for all possible types of the appliance storage.
Unfortunately, additional option for qemu-io is the only and the simplest solution as '-v' creates very specific output, which requires to be parsed. 'qemu-img dd of=/dev/stdout' does not work and the fix would be much more intrusive.
---P.S. Patch to libguestfs will follow.
qemu-io-cmds.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)
qemu-io-cmds.c +13 -4
@@ -718,7 +718,7 @@ static const cmdinfo_t read_cmd = { .cfunc = read_f, .argmin = 2, .argmax = -1, - .args = "[-abCqv] [-P pattern [-s off] [-l len]] off len", + .args = "[-abCqvV] [-P pattern [-s off] [-l len]] off len", .oneline = "reads a number of bytes at a specified offset", .help = read_help, };
@@ -728,6 +728,7 @@ static int read_f(BlockBackend *blk, int argc, char **argv)struct timespec t1, t2;bool Cflag = false, qflag = false, vflag = false;bool Pflag = false, sflag = false, lflag = false, bflag = false; +bool vrawflag = true;int c, cnt, ret;char *buf;int64_t offset;
@@ -737,7 +738,7 @@ static int read_f(BlockBackend *blk, int argc, char **argv)int pattern =0 ;int64_t pattern_offset =0 , pattern_count =0 ; -while ((c =getopt (argc, argv,"bCl:pP:qs:v" )) !=-1 ) { +while ((c =getopt (argc, argv,"bCl:pP:qs:vV" )) !=-1 ) {switch (c) {case 'b' : bflag = true;
@@ -777,6 +778,9 @@ static int read_f(BlockBackend *blk, int argc, char **argv)case 'v' : vflag = true;break ; +case 'V' : + vrawflag = true; +break ;default :qemuio_command_usage (&read_cmd);return -EINVAL ;
@@ -869,10 +873,15 @@ static int read_f(BlockBackend *blk, int argc, char **argv)if (vflag) {dump_buffer (buf, offset, count); } +if (vrawflag) { +write (STDOUT_FILENO , buf, count); + }/* Finally, report back -- -C gives a parsable format */ - t2 =tsub (t2, t1); -print_report ("read" , &t2, offset, count, total, cnt, Cflag); +if (!vrawflag) { + t2 =tsub (t2, t1); +print_report ("read" , &t2, offset, count, total, cnt, Cflag); + } out:qemu_io_free (buf);
Signature · 1 line
2.17.1
Quoted 15 lines · Denis V. Lunev
The problem this patch is trying to address is libguestfs behavior on the appliance startup. It starts supporting to use root=UUID definition in the kernel command line of its root filesystem using
file --/usr/lib64/guestfs/appliance/root
This works fine with RAW image, but we are using QCOW2 as a storage to save a bit of file space and in this case we getQEMU QCOW Image (v3), 1610612736 bytesinstead of UUID of the root filesystem.
The solution is very simple - we should dump first 256k of the image file like the follows
qemu-io -c "read -V 0 256k" appliance | file -
which will provide correct result for all possible types of the appliance storage.
Quoted 4 lines
Unfortunately, additional option for qemu-io is the only and the simplest solution as '-v' creates very specific output, which requires to be parsed. 'qemu-img dd of=/dev/stdout' does not work and the fix would be much more intrusive.
I like the idea of the flag - we could also use it in the nbdkit test suite.
I wonder if the actual flag ('V') is a good idea because I would normally associate that with getting the version of a command. But as these are subcommand flags, that's probably not too relevant here.
So I think this is fine from my point of view.
Rich.
Quoted 72 lines
Signed-off-by Denis V. Lunev <[email protected]>CC Andrey Shinkevich <[email protected]>CC Vladimir Sementsov-Ogievskiy <[email protected]>CC Kevin Wolf <[email protected]>CC Max Reitz <[email protected]>CC Richard W.M. Jones <[email protected]>---
P.S. Patch to libguestfs will follow.qemu-io-cmds.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)qemu-io-cmds.c +13
-4@@ -718,7 +718,7 @@ static const cmdinfo_t read_cmd = { .cfunc = read_f, .argmin = 2, .argmax = -1, - .args = "[-abCqv] [-P pattern [-s off] [-l len]] off len", + .args = "[-abCqvV] [-P pattern [-s off] [-l len]] off len", .oneline = "reads a number of bytes at a specified offset", .help = read_help, };@@ -728,6 +728,7 @@ static int read_f(BlockBackend *blk, int argc, char **argv)struct timespec t1, t2;bool Cflag = false, qflag = false, vflag = false;bool Pflag = false, sflag = false, lflag = false, bflag = false; +bool vrawflag = true;int c, cnt, ret;char *buf;int64_t offset;@@ -737,7 +738,7 @@ static int read_f(BlockBackend *blk, int argc, char **argv)int pattern =0 ;int64_t pattern_offset =0 , pattern_count =0 ; -while ((c =getopt (argc, argv,"bCl:pP:qs:v" )) !=-1 ) { +while ((c =getopt (argc, argv,"bCl:pP:qs:vV" )) !=-1 ) {switch (c) {case 'b' : bflag = true;@@ -777,6 +778,9 @@ static int read_f(BlockBackend *blk, int argc, char **argv)case 'v' : vflag = true;break ; +case 'V' : + vrawflag = true; +break ;default :qemuio_command_usage (&read_cmd);return -EINVAL ;@@ -869,10 +873,15 @@ static int read_f(BlockBackend *blk, int argc, char **argv)if (vflag) {dump_buffer (buf, offset, count); } +if (vrawflag) { +write (STDOUT_FILENO , buf, count); + }/* Finally, report back -- -C gives a parsable format */ - t2 =tsub (t2, t1); -print_report ("read" , &t2, offset, count, total, cnt, Cflag); +if (!vrawflag) { + t2 =tsub (t2, t1); +print_report ("read" , &t2, offset, count, total, cnt, Cflag); + } out:qemu_io_free (buf);Signature · 1 line
2.17.1
Signature · 5 lines
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
Quoted 20 lines · Richard W.M. Jones
Quoted 15 lines · Denis V. Lunev
The problem this patch is trying to address is libguestfs behavior on the appliance startup. It starts supporting to use root=UUID definition in the kernel command line of its root filesystem using
file --/usr/lib64/guestfs/appliance/root
This works fine with RAW image, but we are using QCOW2 as a storage to save a bit of file space and in this case we getQEMU QCOW Image (v3), 1610612736 bytesinstead of UUID of the root filesystem.
The solution is very simple - we should dump first 256k of the image file like the follows
qemu-io -c "read -V 0 256k" appliance | file -
which will provide correct result for all possible types of the appliance storage.Quoted 4 lines
Unfortunately, additional option for qemu-io is the only and the simplest solution as '-v' creates very specific output, which requires to be parsed. 'qemu-img dd of=/dev/stdout' does not work and the fix would be much more intrusive.
Quoted 6 lines
I like the idea of the flag - we could also use it in the nbdkit test suite.
I wonder if the actual flag ('V') is a good idea because I would normally associate that with getting the version of a command. But as these are subcommand flags, that's probably not too relevant here.
we could use '-r' aka --raw and apply it only if -v present.
I think that some opinion from Kevin or Max would be helpful here.
Den
Quoted 81 lines · Denis V. Lunev
The problem this patch is trying to address is libguestfs behavior on the appliance startup. It starts supporting to use root=UUID definition in the kernel command line of its root filesystem using
file --/usr/lib64/guestfs/appliance/root
This works fine with RAW image, but we are using QCOW2 as a storage to save a bit of file space and in this case we getQEMU QCOW Image (v3), 1610612736 bytesinstead of UUID of the root filesystem.
The solution is very simple - we should dump first 256k of the image file like the follows
qemu-io -c "read -V 0 256k" appliance | file -
which will provide correct result for all possible types of the appliance storage.Unfortunately, additional option for qemu-io is the only and the simplest solution as '-v' creates very specific output, which requires to be parsed. 'qemu-img dd of=/dev/stdout' does not work and the fix would be much more intrusive.
Signed-off-by Denis V. Lunev <[email protected]>CC Andrey Shinkevich <[email protected]>CC Vladimir Sementsov-Ogievskiy <[email protected]>CC Kevin Wolf <[email protected]>CC Max Reitz <[email protected]>CC Richard W.M. Jones <[email protected]>---
P.S. Patch to libguestfs will follow.qemu-io-cmds.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)qemu-io-cmds.c +9
-2@@ -718,7 +718,7 @@ static const cmdinfo_t read_cmd = { .cfunc = read_f, .argmin = 2, .argmax = -1, - .args = "[-abCqv] [-P pattern [-s off] [-l len]] off len", + .args = "[-abCqvV] [-P pattern [-s off] [-l len]] off len", .oneline = "reads a number of bytes at a specified offset", .help = read_help, };@@ -728,6 +728,7 @@ static int read_f(BlockBackend *blk, int argc, char **argv)struct timespec t1, t2;bool Cflag = false, qflag = false, vflag = false;bool Pflag = false, sflag = false, lflag = false, bflag = false; +bool vrawflag = true;int c, cnt, ret;char *buf;int64_t offset;@@ -737,7 +738,7 @@ static int read_f(BlockBackend *blk, int argc, char **argv)int pattern =0 ;int64_t pattern_offset =0 , pattern_count =0 ; -while ((c =getopt (argc, argv,"bCl:pP:qs:v" )) !=-1 ) { +while ((c =getopt (argc, argv,"bCl:pP:qs:vV" )) !=-1 ) {switch (c) {case 'b' : bflag = true;@@ -777,6 +778,9 @@ static int read_f(BlockBackend *blk, int argc, char **argv)case 'v' : vflag = true;break ; +case 'V' : + vrawflag = true; +break ;default :qemuio_command_usage (&read_cmd);return -EINVAL ;@@ -869,10 +873,15 @@ static int read_f(BlockBackend *blk, int argc, char **argv)if (vflag) {dump_buffer (buf, offset, count); } +if (vrawflag) { +write (STDOUT_FILENO , buf, count); + }
qemu-io-cmds.c static int read_f(BlockBackend *blk, int argc, char **argv) · line 887
Quoted 10 lines
qemu-io-cmds.c +4
-2/* Finally, report back -- -C gives a parsable format */ - t2 =tsub (t2, t1); -print_report ("read" , &t2, offset, count, total, cnt, Cflag); +if (!vrawflag) { + t2 =tsub (t2, t1); +print_report ("read" , &t2, offset, count, total, cnt, Cflag); + } out:qemu_io_free (buf);
I think -v and -V should be mutually exclusive, as combined output doesn't make real sense. Still, I'm OK with it as is (as well as with -V renamed to -r). I can suggest also -d (aka dump).
Signature · 2 lines
Best regards, Vladimir
Am 10.08.2020 um 16:02 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 84 lines
Quoted 83 lines · Denis V. Lunev
The problem this patch is trying to address is libguestfs behavior on the appliance startup. It starts supporting to use root=UUID definition in the kernel command line of its root filesystem using
file --/usr/lib64/guestfs/appliance/root
This works fine with RAW image, but we are using QCOW2 as a storage to save a bit of file space and in this case we getQEMU QCOW Image (v3), 1610612736 bytesinstead of UUID of the root filesystem.
The solution is very simple - we should dump first 256k of the image file like the follows
qemu-io -c "read -V 0 256k" appliance | file -
which will provide correct result for all possible types of the appliance storage.Unfortunately, additional option for qemu-io is the only and the simplest solution as '-v' creates very specific output, which requires to be parsed. 'qemu-img dd of=/dev/stdout' does not work and the fix would be much more intrusive.
Signed-off-by Denis V. Lunev <[email protected]>CC Andrey Shinkevich <[email protected]>CC Vladimir Sementsov-Ogievskiy <[email protected]>CC Kevin Wolf <[email protected]>CC Max Reitz <[email protected]>CC Richard W.M. Jones <[email protected]>---
P.S. Patch to libguestfs will follow.qemu-io-cmds.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)qemu-io-cmds.c +11
-4@@ -718,7 +718,7 @@ static const cmdinfo_t read_cmd = { .cfunc = read_f, .argmin = 2, .argmax = -1, - .args = "[-abCqv] [-P pattern [-s off] [-l len]] off len", + .args = "[-abCqvV] [-P pattern [-s off] [-l len]] off len", .oneline = "reads a number of bytes at a specified offset", .help = read_help, };@@ -728,6 +728,7 @@ static int read_f(BlockBackend *blk, int argc, char **argv)struct timespec t1, t2;bool Cflag = false, qflag = false, vflag = false;bool Pflag = false, sflag = false, lflag = false, bflag = false; +bool vrawflag = true;int c, cnt, ret;char *buf;int64_t offset;@@ -737,7 +738,7 @@ static int read_f(BlockBackend *blk, int argc, char **argv)int pattern =0 ;int64_t pattern_offset =0 , pattern_count =0 ; -while ((c =getopt (argc, argv,"bCl:pP:qs:v" )) !=-1 ) { +while ((c =getopt (argc, argv,"bCl:pP:qs:vV" )) !=-1 ) {switch (c) {case 'b' : bflag = true;@@ -777,6 +778,9 @@ static int read_f(BlockBackend *blk, int argc, char **argv)case 'v' : vflag = true;break ; +case 'V' : + vrawflag = true; +break ;default :qemuio_command_usage (&read_cmd);return -EINVAL ;@@ -869,10 +873,15 @@ static int read_f(BlockBackend *blk, int argc, char **argv)if (vflag) {dump_buffer (buf, offset, count); } +if (vrawflag) { +write (STDOUT_FILENO , buf, count); + }/* Finally, report back -- -C gives a parsable format */ - t2 =tsub (t2, t1); -print_report ("read" , &t2, offset, count, total, cnt, Cflag); +if (!vrawflag) { + t2 =tsub (t2, t1);
Quoted 8 lines
qemu-io-cmds.c static int read_f(BlockBackend *blk, int argc, char **argv)
Quoted 5 lines
qemu-io-cmds.c +2
-0+print_report ("read" , &t2, offset, count, total, cnt, Cflag); + } out:qemu_io_free (buf);I think -v and -V should be mutually exclusive, as combined output doesn't make real sense. Still, I'm OK with it as is (as well as with -V renamed to -r). I can suggest also -d (aka dump).
I like -d (maybe with an alias --dump), though in the end the naming is secondary. I think having some flag like this is very useful.
How about adding the same thing to write, i.e. get the buffer to write from stdin?
Kevin
Quoted 92 lines · Kevin Wolf
Am 10.08.2020 um 16:02 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 83 lines
Quoted 82 lines · Denis V. Lunev
The problem this patch is trying to address is libguestfs behavior on the appliance startup. It starts supporting to use root=UUID definition in the kernel command line of its root filesystem using
file --/usr/lib64/guestfs/appliance/root
This works fine with RAW image, but we are using QCOW2 as a storage to save a bit of file space and in this case we getQEMU QCOW Image (v3), 1610612736 bytesinstead of UUID of the root filesystem.
The solution is very simple - we should dump first 256k of the image file like the follows
qemu-io -c "read -V 0 256k" appliance | file -
which will provide correct result for all possible types of the appliance storage.Unfortunately, additional option for qemu-io is the only and the simplest solution as '-v' creates very specific output, which requires to be parsed. 'qemu-img dd of=/dev/stdout' does not work and the fix would be much more intrusive.
Signed-off-by Denis V. Lunev <[email protected]>CC Andrey Shinkevich <[email protected]>CC Vladimir Sementsov-Ogievskiy <[email protected]>CC Kevin Wolf <[email protected]>CC Max Reitz <[email protected]>CC Richard W.M. Jones <[email protected]>---
P.S. Patch to libguestfs will follow.qemu-io-cmds.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-)qemu-io-cmds.c +10
-4@@ -718,7 +718,7 @@ static const cmdinfo_t read_cmd = { .cfunc = read_f, .argmin = 2, .argmax = -1, - .args = "[-abCqv] [-P pattern [-s off] [-l len]] off len", + .args = "[-abCqvV] [-P pattern [-s off] [-l len]] off len", .oneline = "reads a number of bytes at a specified offset", .help = read_help, };@@ -728,6 +728,7 @@ static int read_f(BlockBackend *blk, int argc, char **argv)struct timespec t1, t2;bool Cflag = false, qflag = false, vflag = false;bool Pflag = false, sflag = false, lflag = false, bflag = false; +bool vrawflag = true;int c, cnt, ret;char *buf;int64_t offset;@@ -737,7 +738,7 @@ static int read_f(BlockBackend *blk, int argc, char **argv)int pattern =0 ;int64_t pattern_offset =0 , pattern_count =0 ; -while ((c =getopt (argc, argv,"bCl:pP:qs:v" )) !=-1 ) { +while ((c =getopt (argc, argv,"bCl:pP:qs:vV" )) !=-1 ) {switch (c) {case 'b' : bflag = true;@@ -777,6 +778,9 @@ static int read_f(BlockBackend *blk, int argc, char **argv)case 'v' : vflag = true;break ; +case 'V' : + vrawflag = true; +break ;default :qemuio_command_usage (&read_cmd);return -EINVAL ;@@ -869,10 +873,15 @@ static int read_f(BlockBackend *blk, int argc, char **argv)if (vflag) {dump_buffer (buf, offset, count); } +if (vrawflag) { +write (STDOUT_FILENO , buf, count); + }/* Finally, report back -- -C gives a parsable format */ - t2 =tsub (t2, t1); -print_report ("read" , &t2, offset, count, total, cnt, Cflag); +if (!vrawflag) {Quoted 8 lines
qemu-io-cmds.c static int read_f(BlockBackend *blk, int argc, char **argv)
Quoted 6 lines
qemu-io-cmds.c +3
-0+ t2 =tsub (t2, t1); +print_report ("read" , &t2, offset, count, total, cnt, Cflag); + } out:qemu_io_free (buf);I think -v and -V should be mutually exclusive, as combined output doesn't make real sense. Still, I'm OK with it as is (as well as with -V renamed to -r). I can suggest also -d (aka dump).
Quoted 5 lines
I like -d (maybe with an alias --dump), though in the end the naming is secondary. I think having some flag like this is very useful.
How about adding the same thing to write, i.e. get the buffer to write from stdin?
no prob, will do :)
Den
Quoted 6 lines · Denis V. Lunev
The problem this patch is trying to address is libguestfs behavior on the appliance startup. It starts supporting to use root=UUID definition in the kernel command line of its root filesystem using
file --/usr/lib64/guestfs/appliance/root
This works fine with RAW image, but we are using QCOW2 as a storage to save a bit of file space and in this case we get
Quoted 8 lines
QEMU QCOW Image (v3), 1610612736 bytesinstead of UUID of the root filesystem.
The solution is very simple - we should dump first 256k of the image file like the follows
qemu-io -c "read -V 0 256k" appliance | file -
which will provide correct result for all possible types of the appliance storage.
Is 'appliance' a qcow2 file? If so, another way to accomplish this could include:
nbdkit streaming --run 'qemu-img convert --image-opts driver=raw,size=256,file.driver=qcow2,file.file.driver=file,file.file.filename=appliance "$uri"' | file -
which says to use qemu-img to open a length-limited view of the file, and stream the entire thing to an NBD server, where nbdkit then provides a way to convert that to a pipeline to feed into file (since qemu's NBD server doesn't directly like writing into a pipe).
I'm also wondering if the 'nbdcopy' program (not yet released in a stable version, but available in libnbd.git) could be put to use on this front, with some way to quickly combine it with qemu-nbd serving appliance.qcow2.
Quoted 5 lines
Unfortunately, additional option for qemu-io is the only and the simplest solution as '-v' creates very specific output, which requires to be parsed. 'qemu-img dd of=/dev/stdout' does not work and the fix would be much more intrusive.
'qemu-img dd' _should_ be merely syntactic sugar for 'qemu-img convert', but it isn't yet. Until we rectify that feature parity (there are things that dd can't do like better output control and skipping, and there are things that convert can't do like length limiting and offset selection), hacking up 'qemu-io' (which is for debugging use only) is a tolerable short-term solution; but qemu-io was NEVER intended for stable use cases. If adding this makes debugging qemu easier, then go for it; but if the real problem is that qemu-img is missing functionality, we should instead be focusing on fixing qemu-img.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Quoted 16 lines · Eric Blake
Quoted 9 lines · Denis V. Lunev
The problem this patch is trying to address is libguestfs behavior on the appliance startup. It starts supporting to use root=UUID definition in the kernel command line of its root filesystem using
file --/usr/lib64/guestfs/appliance/root
This works fine with RAW image, but we are using QCOW2 as a storage to save a bit of file space and in this case we get
QEMU QCOW Image (v3), 1610612736 bytes
instead of UUID of the root filesystem.Quoted 5 lines
The solution is very simple - we should dump first 256k of the image file like the follows
qemu-io -c "read -V 0 256k" appliance | file -
which will provide correct result for all possible types of the appliance storage.
Quoted 3 lines
Is 'appliance' a qcow2 file? If so, another way to accomplish this could include:
nbdkit streaming --run 'qemu-img convert --image-opts driver=raw,size=256,file.driver=qcow2,file.file.driver=file,file.file.filename=appliance "$uri"' | file -
This adds a dependency on nbdkit streaming plugin, not installed by default in our RH-based distributive. Probably the simplest thing is just create a temporary file for qemu-img convert (or dd) target, and then call file command on it.
Quoted 6 lines
which says to use qemu-img to open a length-limited view of the file, and stream the entire thing to an NBD server, where nbdkit then provides a way to convert that to a pipeline to feed into file (since qemu's NBD server doesn't directly like writing into a pipe).
I'm also wondering if the 'nbdcopy' program (not yet released in a stable version, but available in libnbd.git) could be put to use on this front, with some way to quickly combine it with qemu-nbd serving appliance.qcow2.
Quoted 1 line
Quoted 7 lines
Quoted 4 lines
Unfortunately, additional option for qemu-io is the only and the simplest solution as '-v' creates very specific output, which requires to be parsed. 'qemu-img dd of=/dev/stdout' does not work and the fix would be much more intrusive.
'qemu-img dd' _should_ be merely syntactic sugar for 'qemu-img convert', but it isn't yet. Until we rectify that feature parity (there are things that dd can't do like better output control and skipping, and there are things that convert can't do like length limiting and offset selection), hacking up 'qemu-io' (which is for debugging use only) is a tolerable short-term solution; but qemu-io was NEVER intended for stable use cases. If adding this makes debugging qemu easier, then go for it; but if the real problem is that qemu-img is missing functionality, we should instead be focusing on fixing qemu-img.
Thanks for clarifying this!
If we are going to improve qemu-img at some point, I'm not sure that convert or dd is correct places to do it, as they want target to be a block node. But how to create a block-node on top of /dev/stdout? Detect it and behave another way in img_dd function? Or may be better create another command qemu-img dump, for dumping the contents to stdout?
Signature · 2 lines
Best regards, Vladimir