Parameterise self-announce
cover Parameterise self-announce
From: "Dr. David Alan Gilbert" <[email protected]>
The RARP/ARP announce mechanisms have a bunch of fixed timings/counts and on a system with a sluggish network the whole announce process can be a bit short, ending before the networking is ready. Make all these fixed values parameterised, document it and add a basic test.
The default behaviour is unchanged, that is we have:
<p> 50ms <p> 150ms <p> 250ms <p> 350ms <p>
Dave
Dr. David Alan Gilbert (6):
migration/announce: Add parameters for self-announce migration/announce: Accessor functions for parameters migration/announce: Use the new parameters migration/announce: Update hmp migrate parameter info/set migration/announce: Document self-announce mechanism migration/announce: Add test
docs/migration.txt | 40 ++++++++++ hmp.c | 36 +++++++++ hw/net/virtio-net.c | 3 +- include/migration/migration.h | 6 ++ include/migration/vmstate.h | 10 --- migration/migration.c | 72 ++++++++++++++++++ migration/savevm.c | 32 +++++++- qapi-schema.json | 34 ++++++++- tests/Makefile.include | 2 + tests/announce-test.c | 165 ++++++++++++++++++++++++++++++++++++++++++ 10 files changed, 385 insertions(+), 15 deletions(-) create mode 100644 tests/announce-test.c
Signature · 1 line
2.9.3
PATCH 1/6 migration/announce: Add parameters for self-announce
From: "Dr. David Alan Gilbert" <[email protected]>
Add the parameters and wire into the set/get routines.
--- migration/migration.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++ qapi-schema.json | 34 +++++++++++++++++++++++++++++++-- 2 files changed, 84 insertions(+), 2 deletions(-)
migration/migration.c +52 -0
@@ -53,6 +53,14 @@#define MAX_MIGRATE_DOWNTIME_SECONDS 2000#define MAX_MIGRATE_DOWNTIME (MAX_MIGRATE_DOWNTIME_SECONDS * 1000) +/* Parameters for self_announce_delay giving a stream of RARP/ARP +* packets after migration. +*/ +#define DEFAULT_MIGRATE_ANNOUNCE_INITIAL 50 +#define DEFAULT_MIGRATE_ANNOUNCE_MAX 550 +#define DEFAULT_MIGRATE_ANNOUNCE_ROUNDS 5 +#define DEFAULT_MIGRATE_ANNOUNCE_STEP 100 +/* Default compression thread count */ #define DEFAULT_MIGRATE_COMPRESS_THREAD_COUNT 8/* Default decompression thread count, usually decompression is at
@@ -97,6 +105,10 @@ MigrationState *migrate_get_current(void) .xbzrle_cache_size = DEFAULT_MIGRATE_CACHE_SIZE, .mbps = -1, .parameters = { + .announce_initial = DEFAULT_MIGRATE_ANNOUNCE_INITIAL, + .announce_max = DEFAULT_MIGRATE_ANNOUNCE_MAX, + .announce_rounds = DEFAULT_MIGRATE_ANNOUNCE_ROUNDS, + .announce_step = DEFAULT_MIGRATE_ANNOUNCE_STEP, .compress_level = DEFAULT_MIGRATE_COMPRESS_LEVEL, .compress_threads = DEFAULT_MIGRATE_COMPRESS_THREAD_COUNT, .decompress_threads = DEFAULT_MIGRATE_DECOMPRESS_THREAD_COUNT,
@@ -580,6 +592,14 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)MigrationState *s =migrate_get_current (); params =g_malloc0 (sizeof (*params)); + params->has_announce_initial = true; + params->announce_initial = s->parameters.announce_initial; + params->has_announce_max = true; + params->announce_max = s->parameters.announce_max; + params->has_announce_rounds = true; + params->announce_rounds = s->parameters.announce_rounds; + params->has_announce_step = true; + params->announce_step = s->parameters.announce_step; params->has_compress_level = true; params->compress_level = s->parameters.compress_level; params->has_compress_threads = true;
@@ -809,6 +829,26 @@ void qmp_migrate_set_parameters(MigrationParameters *params, Error **errp) {MigrationState *s =migrate_get_current (); +if (params->has_announce_initial && + (params->announce_initial <1 || params->announce_initial >100000 )) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"announce_initial" , +"is invalid, it should be in the range of 1 to 100000 ms" ); + } +if (params->has_announce_max && + (params->announce_max <1 || params->announce_max >100000 )) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"announce_max" , +"is invalid, it should be in the range of 1 to 100000 ms" ); + } +if (params->has_announce_rounds && + (params->announce_rounds <1 || params->announce_rounds >1000 )) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"announce_rounds" , +"is invalid, it should be in the range of 1 to 1000" ); + } +if (params->has_announce_step && + (params->announce_step <0 || params->announce_step >10000 )) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"announce_step" , +"is invalid, it should be in the range of 1 to 10000 ms" ); + }if (params->has_compress_level && (params->compress_level <0 || params->compress_level >9 )) { error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "compress_level",
@@ -865,6 +905,18 @@ void qmp_migrate_set_parameters(MigrationParameters *params, Error **errp) "is invalid, it should be positive"); } +if (params->has_announce_initial) { + s->parameters.announce_initial = params->announce_initial; + } +if (params->has_announce_max) { + s->parameters.announce_max = params->announce_max; + } +if (params->has_announce_rounds) { + s->parameters.announce_rounds = params->announce_rounds; + } +if (params->has_announce_step) { + s->parameters.announce_step = params->announce_step; + }if (params->has_compress_level) { s->parameters.compress_level = params->compress_level; }
qapi-schema.json +32 -2
@@ -960,6 +960,18 @@ # # Migration parameters enumeration # +# @announce-initial: The inital delay (in ms) before sending the first announce +# (Since 2.10) +# +# @announce-max: The maximum delay (in ms) between packets in the announcment +# (Since 2.10) +# +# @announce-rounds: The number of self-announce packets sent after migration +# (Since 2.10) +# +# @announce-step: The increase in delay (in ms) between subsequent packets in +# the announcement (Since 2.10) +# # @compress-level: Set the compression level to be used in live migration, # the compression level is an integer between 0 and 9, where 0 means # no compression, 1 means the best compression speed, and 9 means best
@@ -1009,7 +1021,9 @@ # Since: 2.4 ## { 'enum': 'MigrationParameter', - 'data': ['compress-level', 'compress-threads', 'decompress-threads', + 'data': ['announce-initial', 'announce-max', + 'announce-rounds', 'announce-step', + 'compress-level', 'compress-threads', 'decompress-threads', 'cpu-throttle-initial', 'cpu-throttle-increment', 'tls-creds', 'tls-hostname', 'max-bandwidth', 'downtime-limit', 'x-checkpoint-delay' ] }
@@ -1038,6 +1052,18 @@ # ('query-migrate-parameters'), with the exception of tls-creds and # tls-hostname. # +# @announce-initial: The inital delay (in ms) before sending the first announce +# (Since 2.10) +# +# @announce-max: The maximum delay (in ms) between packets in the announcment +# (Since 2.10) +# +# @announce-rounds: The number of self-announce packets sent after migration +# (Since 2.10) +# +# @announce-step: The increase in delay (in ms) between subsequent packets in +# the announcement (Since 2.10) +# # @compress-level: compression level # # @compress-threads: compression thread count
@@ -1082,7 +1108,11 @@ # Since: 2.4 ## { 'struct': 'MigrationParameters', - 'data': { '*compress-level': 'int', + 'data': { '*announce-initial': 'int', + '*announce-max': 'int', + '*announce-rounds': 'int', + '*announce-step': 'int', + '*compress-level': 'int', '*compress-threads': 'int', '*decompress-threads': 'int', '*cpu-throttle-initial': 'int',
Signature · 1 line
2.9.3
Re: [Qemu-devel] [PATCH 1/6] migration/announce: Add parameters for self-announce
Quoted 12 lines · Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" <[email protected]>
Add the parameters and wire into the set/get routines.
Signed-off-by Dr. David Alan Gilbert <[email protected]>--- migration/migration.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++ qapi-schema.json | 34 +++++++++++++++++++++++++++++++-- 2 files changed, 84 insertions(+), 2 deletions(-)
migration/migration.c void qmp_migrate_set_parameters(MigrationParameters *params, Error **errp) · line 838
Quoted 7 lines
migration/migration.c +7
-0+if (params->has_announce_initial && + (params->announce_initial <1 || params->announce_initial >100000 )) { +error_setg (errp,QERR_INVALID_PARAMETER_VALUE ,"announce_initial" , +"is invalid, it should be in the range of 1 to 100000 ms" ); + } +if (params->has_announce_max && + (params->announce_max <1 || params->announce_max >100000 )) {
You're doing range validation here...
qapi-schema.json line 964
Quoted 7 lines
+++
b/qapi-schema.jsonqapi-schema.json +2
-0@@ -960,6 +960,18 @@ # # Migration parameters enumeration # +# @announce-initial: The inital delay (in ms) before sending the first announce +# (Since 2.10)
...so it would be useful to mention the bounds here. Also, mentioning the defaults is a good idea.
s/inital/initial/
qapi-schema.json line 967
Quoted 3 lines
qapi-schema.json +3
-0+# +# @announce-max: The maximum delay (in ms) between packets in the announcment +# (Since 2.10)
s/announcment/announcement/
Quoted 9 lines
qapi-schema.json +7
-0+# +# @announce-rounds: The number of self-announce packets sent after migration +# (Since 2.10) +# +# @announce-step: The increase in delay (in ms) between subsequent packets in +# the announcement (Since 2.10) +#
qapi-schema.json line 1059
Quoted 8 lines
qapi-schema.json +5
-0@@ -1038,6 +1052,18 @@ # ('query-migrate-parameters'), with the exception of tls-creds and # tls-hostname. # +# @announce-initial: The inital delay (in ms) before sending the first announce +# (Since 2.10) +# +# @announce-max: The maximum delay (in ms) between packets in the announcment +# (Since 2.10)
same two typos
Signature · 2 lines
Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
PATCH 2/6 migration/announce: Accessor functions for parameters
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Standard accessor functions for each of the parameters
--- include/migration/migration.h | 5 +++++ migration/migration.c | 20 ++++++++++++++++++++ 2 files changed, 25 insertions(+)
include/migration/migration.h +5 -0
@@ -340,6 +340,11 @@ int migrate_compress_threads(void);int migrate_decompress_threads (void );bool migrate_use_events (void ); +int migrate_announce_initial (void ); +int migrate_announce_max (void ); +int migrate_announce_rounds (void ); +int migrate_announce_step (void ); +/* Sending on the return path - generic and then for each message type */ void migrate_send_rp_message(MigrationIncomingState *mis, enum mig_rp_message_type message_type,
migration/migration.c +20 -0
@@ -1483,6 +1483,26 @@ int64_t migrate_xbzrle_cache_size(void)return s->xbzrle_cache_size; } +int migrate_announce_initial (void ) +{ +return migrate_get_current ()->parameters.announce_initial; +} + +int migrate_announce_max (void ) +{ +return migrate_get_current ()->parameters.announce_max; +} + +int migrate_announce_rounds (void ) +{ +return migrate_get_current ()->parameters.announce_rounds; +} + +int migrate_announce_step (void ) +{ +return migrate_get_current ()->parameters.announce_step; +} +/* migration thread support */ /* * Something bad happened to the RP stream, mark an error
Signature · 1 line
2.9.3
PATCH 3/6 migration/announce: Use the new parameters
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Rework the existing constants to use the new parameters.
--- hw/net/virtio-net.c | 3 ++- include/migration/migration.h | 1 + include/migration/vmstate.h | 10 ---------- migration/savevm.c | 32 ++++++++++++++++++++++++++++++-- 4 files changed, 33 insertions(+), 13 deletions(-)
hw/net/virtio-net.c +2 -1
@@ -19,6 +19,7 @@#include "net/tap.h" #include "qemu/error-report.h" #include "qemu/timer.h" +#include "migration/migration.h" #include "hw/virtio/virtio-net.h" #include "net/vhost_net.h" #include "hw/virtio/virtio-bus.h"
@@ -1611,7 +1612,7 @@ static int virtio_net_post_load_device(void *opaque, int version_id)if (virtio_vdev_has_feature (vdev,VIRTIO_NET_F_GUEST_ANNOUNCE ) &&virtio_vdev_has_feature (vdev,VIRTIO_NET_F_CTRL_VQ )) { - n->announce_counter =SELF_ANNOUNCE_ROUNDS ; + n->announce_counter =migrate_announce_rounds ();timer_mod (n->announce_timer,qemu_clock_get_ms (QEMU_CLOCK_VIRTUAL )); }
include/migration/migration.h +1 -0
@@ -344,6 +344,7 @@ int migrate_announce_initial(void);int migrate_announce_max (void );int migrate_announce_rounds (void );int migrate_announce_step (void ); +int64_t self_announce_delay (int round);/* Sending on the return path - generic and then for each message type */ void migrate_send_rp_message(MigrationIncomingState *mis,
include/migration/vmstate.h +0 -10
@@ -1003,8 +1003,6 @@ extern const VMStateInfo vmstate_info_qtailq;#define VMSTATE_END_OF_LIST () \ {} -#define SELF_ANNOUNCE_ROUNDS 5 -void loadvm_free_handlers (MigrationIncomingState *mis); int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
@@ -1038,14 +1036,6 @@ void vmstate_register_ram(struct MemoryRegion *memory, DeviceState *dev);void vmstate_unregister_ram (struct MemoryRegion *memory,DeviceState *dev);void vmstate_register_ram_global (struct MemoryRegion *memory); -static inline -int64_t self_announce_delay (int round) -{ -assert (round <SELF_ANNOUNCE_ROUNDS && round >0 ); -/* delay 50ms, 150ms, 250ms, ... */ -return 50 + (SELF_ANNOUNCE_ROUNDS - round -1 ) *100 ; -} -void dump_vmstate_json_to_file (FILE *out_fp);#endif
migration/savevm.c +30 -2
@@ -118,16 +118,44 @@ static void qemu_announce_self_iter(NICState *nic, void *opaque)qemu_send_packet_raw (qemu_get_queue (nic), buf, len); } +/* Increasing delays between packets in the network announcments, +* both in the RARPs sent by QEMU and the ARPs triggered via virtio-net +* for the guest to send. +* +* Default parameters generate delays of +* 50, 150, 250, 350 +* between 5 packets. +* which corresponds to: +* <> initial <> initial+step <> initial+2*step <> initial+3*step <> +* between 'rounds' packets +* A maximum can be used to override the step after a few packets. +*/ +int64_t self_announce_delay (int round) +{ +int64_t ret; + ret =migrate_announce_initial () + + (migrate_announce_rounds () - round -1 ) * +migrate_announce_step (); +if (ret <0 || ret >migrate_announce_max ()) { + ret =migrate_announce_max (); + } +return ret; +}static void qemu_announce_self_once (void *opaque) { -static int count =SELF_ANNOUNCE_ROUNDS ; +static bool once = true; +static int count =-1 ;QEMUTimer *timer = *(QEMUTimer **)opaque; +if (once) { + count =migrate_announce_rounds (); + once = false; + } +qemu_foreach_nic (qemu_announce_self_iter,NULL );if (--count) { -/* delay 50ms, 150ms, 250ms, ... */ timer_mod (timer,qemu_clock_get_ms (QEMU_CLOCK_REALTIME ) +self_announce_delay (count)); }else {
Signature · 1 line
2.9.3
PATCH 4/6 migration/announce: Update hmp migrate parameter info/set
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add the new announce parameters to HMPs migrate_parameter commands.
--- hmp.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
hmp.c +36 -0
@@ -285,6 +285,22 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)if (params) {monitor_printf (mon,"parameters:" ); +assert (params->has_announce_initial); +monitor_printf (mon," %s: %" PRId64, + MigrationParameter_lookup[MIGRATION_PARAMETER_ANNOUNCE_INITIAL ], + params->announce_initial); +assert (params->has_announce_max); +monitor_printf (mon," %s: %" PRId64, + MigrationParameter_lookup[MIGRATION_PARAMETER_ANNOUNCE_MAX ], + params->announce_max); +assert (params->has_announce_rounds); +monitor_printf (mon," %s: %" PRId64, + MigrationParameter_lookup[MIGRATION_PARAMETER_ANNOUNCE_ROUNDS ], + params->announce_rounds); +assert (params->has_announce_step); +monitor_printf (mon," %s: %" PRId64, + MigrationParameter_lookup[MIGRATION_PARAMETER_ANNOUNCE_STEP ], + params->announce_step);assert (params->has_compress_level); monitor_printf(mon, " %s: %" PRId64, MigrationParameter_lookup[MIGRATION_PARAMETER_COMPRESS_LEVEL],
@@ -1354,6 +1370,22 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)if (strcmp (param, MigrationParameter_lookup[i]) ==0 ) {MigrationParameters p = {0 };switch (i) { +case MIGRATION_PARAMETER_ANNOUNCE_INITIAL : + p.has_announce_initial = true; + use_int_value = true; +break ; +case MIGRATION_PARAMETER_ANNOUNCE_MAX : + p.has_announce_max = true; + use_int_value = true; +break ; +case MIGRATION_PARAMETER_ANNOUNCE_ROUNDS : + p.has_announce_rounds = true; + use_int_value = true; +break ; +case MIGRATION_PARAMETER_ANNOUNCE_STEP : + p.has_announce_step = true; + use_int_value = true; +break ;case MIGRATION_PARAMETER_COMPRESS_LEVEL : p.has_compress_level = true; use_int_value = true;
@@ -1410,6 +1442,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) }/* Set all integers; only one has_FOO will be set, and * the code ignores the remaining values */ + p.announce_initial = valueint; + p.announce_max = valueint; + p.announce_rounds = valueint; + p.announce_step = valueint; p.compress_level = valueint; p.compress_threads = valueint; p.decompress_threads = valueint;
Signature · 1 line
2.9.3
PATCH 5/6 migration/announce: Document self-announce mechanism
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Explain the timings and differences between the ARP and RARP streams we send.
--- docs/migration.txt | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+)
docs/migration.txt +40 -0
@@ -553,3 +553,43 @@ Postcopy now works with hugetlbfs backed memory: hugepages works well, however 1GB hugepages are likely to be problematic since it takes ~1 second to transfer a 1GB hugepage across a 10Gbps link, and until the full page is transferred the destination thread is blocked. + +== Network self announce == + +At the end of a migration QEMU sends a sequence of packets to 'announce' the VMs +arrival on the new host in the hope that the L2 network will notice and start +sending the packets to it. This, 'self-announce' is split into two parts that +happens at slightly different times: + + a) RARP - a RARP sequence is sent when the last of the device data has been + received but before the guest CPUs are started. Note that when -S is used + to stop QEMU starting the CPUs automatically, the RARPs are still sent + at the point where the device data is received and the guest CPUs are ready + to be started. + + b) ARPs - for guests with virtio NICs and drivers new enough to support the + GUEST_ANNOUNCE feature, the guests are told to emit gratuitous ARPs just + after the guest CPUs are started after an incoming migration. Note the + packets are generated by the guest itself and have the advantage of knowing + the guests IP address. + +The two sets may overlap. + +In both cases a stream of packets are sent, the number and timing can be +configured using the announce* migration parameters, which must be set prior +to the end of the incoming migration. + + <P> di <P> di+ds <P> di+2*ds <P> ..... <P> + + 'announce-rounds' packets are sent. + The gap between the 1st and 2nd packet is 'announce-initial'. + The gap between the 2nd and 3rd packet is increased by 'announce-step' + A maximum can be set such that the gap between subsequent packets stops + increasing. + + The default is: + <P> 50ms <P> 150ms <P> 250ms <P> 350ms <P> + +Some complex network systems may take longer than this to reconfigure +and thus it may be useful to increase the count or delays. +
Signature · 1 line
2.9.3
PATCH 6/6 migration/announce: Add test
In reply to Dr. David Alan Gilbert (git)From: "Dr. David Alan Gilbert" <[email protected]>
Add a test for the basic RARP announce. Testing the ARP would need a reasonably complex guest.
--- tests/Makefile.include | 2 + tests/announce-test.c | 165 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 167 insertions(+) create mode 100644 tests/announce-test.c
tests/Makefile.include +2 -0
@@ -258,6 +258,7 @@ endif check-qtest-i386-y += tests/test-netfilter$(EXESUF) check-qtest-i386-y += tests/test-filter-mirror$(EXESUF) check-qtest-i386-y += tests/test-filter-redirector$(EXESUF) +check-qtest-i386-y += tests/announce-test$(EXESUF) check-qtest-i386-y += tests/postcopy-test$(EXESUF) check-qtest-i386-y += tests/test-x86-cpuid-compat$(EXESUF) check-qtest-x86_64-y += $(check-qtest-i386-y)
@@ -285,6 +286,7 @@ check-qtest-ppc64-y += tests/boot-order-test$(EXESUF) check-qtest-ppc64-y += tests/prom-env-test$(EXESUF) check-qtest-ppc64-y += tests/pnv-xscom-test$(EXESUF) check-qtest-ppc64-y += tests/drive_del-test$(EXESUF) +check-qtest-ppc64-y += tests/announce-test$(EXESUF) check-qtest-ppc64-y += tests/postcopy-test$(EXESUF) check-qtest-ppc64-y += tests/boot-serial-test$(EXESUF) check-qtest-ppc64-y += tests/rtas-test$(EXESUF)
tests/announce-test.c new file +165 -0
165 lines not on this page · raw message
Signature · 1 line
2.9.3