Skip to contentSkip to conversation

Live migration sequence

Qemu-devel Pavel Fedin 3 participants 9 messages Oct 8, 2015 – Oct 16, 2015 KVM arm64

lore
Share & tools
lore
#0 lore raw

Hello!

I would like to clarify, what is the exact live migration sequence in qemu?

I mean - there are pre_save and post_load callbacks for VMState structures. Is there any determined
order of calling them related to memory contents migration? In other words, is there any guarantee that pre_save is called before RAM migrates, and post_load is called after RAM migrates?
The answer to this question is important for developing vITS live migration, where i have to dump
internal ITS state into in-memory tables before the migration starts, and then get it back in cache on destination.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

#1 lore raw
Quoted 7 lines · * Pavel Fedin ([email protected]) wrote:

Hello!

I would like to clarify, what is the exact live migration sequence in qemu?

I mean - there are pre_save and post_load callbacks for VMState structures. Is there any determined
order of calling them related to memory contents migration? In other words, is there any guarantee that pre_save is called before RAM migrates, and post_load is called after RAM migrates?

The pre_load/pre_save and post_load relate to the particular VMState the functions are attached to; so if you use them on a VMState of a particular device the only thing you know is that the pre_save is called just before the system writes the description out; and on loading the pre_load is called just before it reads the data, and post_load just after it's read the data.

Ordering relating to RAM is a separate question; in general RAM is normally loaded before all of the non-iterative devices.

Quoted 3 lines

The answer to this question is important for developing vITS live migration, where i have to dump
internal ITS state into in-memory tables before the migration starts, and then get it back in cache on destination.

What's an ITS ?
With a related question, how big are the tables and can it change during the iterated part of the migrate?

Dave

Quoted 9 lines

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#2 lore raw

Hello!

Sorry for the delayed reply.

Quoted 1 line

What's an ITS ?

Interrupt Translation Service. In a short, it's a thing responsible for handling PCIe MSI-X
interrupts on ARM64 architecture.

Quoted 2 lines

With a related question, how big are the tables and can it change during the iterated part of the migrate?

Tables are something like 64K each. They hold mappings between device/event IDs and actual IRQ
numbers.
Unfortunately i don't know how to answer the second part of the question, about iterated part. Can
you explain in details, what is it and how does it work?
Or, well, we could put the question the other way: imagine that in pre_save i tell my emulated
device to flush its cached state into RAM-based tables. In post_load i could tell the device to re-read data from RAM into its cache. So, what do i need in order to make these tables in RAM to migrate correctly?

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

#3 lore raw
Quoted 8 lines · * Pavel Fedin ([email protected]) wrote:

Hello!

Sorry for the delayed reply.

Quoted 1 line

What's an ITS ?

Interrupt Translation Service. In a short, it's a thing responsible for handling PCIe MSI-X
interrupts on ARM64 architecture.

OK; I asked Peter (cc'd) to explain a bit more about the ITS to me.

Quoted 8 lines
Quoted 2 lines

With a related question, how big are the tables and can it change during the iterated part of the migrate?

Tables are something like 64K each. They hold mappings between device/event IDs and actual IRQ
numbers.

Unfortunately i don't know how to answer the second part of the question, about iterated part. Can
you explain in details, what is it and how does it work?

QEMU migrates stuff in two different ways:
a) Like a device where at the end of the migration it transmits all the information
b) Like RAM, (iterated) - in this it sends the contents across but does this while the guest is still running, changes that are made to the RAM are then transmitted over and over again until the amount of changed RAM is small; then we stop the guest, transmit those last few changes and then do (a).

Quoted 4 lines

Or, well, we could put the question the other way: imagine that in pre_save i tell my emulated
device to flush its cached state into RAM-based tables. In post_load i could tell the device to re-read data from RAM into its cache. So, what do i need in order to make these tables in RAM to migrate correctly?

At 64k that's pretty small; however Peter explained to me that's per-cpu so that potentially could be huge. If it was only small I'd do it just like a device which is nice and simple; but since it can get big then it's going to be more interesting, and since it's part of guest RAM you need to be careful.

The pre_load/post_load are all relative to a particular device; they're not hooked around when other stuff (RAM) gets migrated; it sounds like you need another hook.

If I understand correctly what you need is to find a hook to dump the state into guest ram, but then you also need to keep the state updated in guest RAM during the migration.

Some thoughts:
a) There is a migration state notifier list - see add_migration_state_change_notifier (spice calls it)
- but I don't think it's called in the right places for your needs; we
could add some more places that gets called.

b) Once you're in the device state saving (a above) you must not change guest RAM, because at that point the migration code won't send any new changes across to the destination. So any sync's you're going to do have to happen before/at the time we stop the CPU and do the final RAM sync. On the plus side, when you're loading the device state in (a) you can be sure the RAM contents are there.

c) Watch out for the size of that final sync; if you have lots of these ITS and they all update their 64k page at the point we stop the CPU then you're going to generate a lot of RAM that needs syncing.

Dave

Quoted 7 lines

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#4 lore raw

Hello!

Quoted 5 lines

b) Once you're in the device state saving (a above) you must not change guest RAM, because at that point the migration code won't send any new changes across to the destination. So any sync's you're going to do have to happen before/at the time we stop the CPU and do the final RAM sync. On the plus side, when you're loading the device state in (a) you can be sure the RAM contents are there.

This is good. I think, in this case i can teach the kernel (here we talk about accelerated
in-kernel irqchip implementation) to flush ITS caches when a CPU is stopped. This will do the job.

Quoted 3 lines

c) Watch out for the size of that final sync; if you have lots of these ITS and they all update their 64k page at the point we stop the CPU then you're going to generate a lot of RAM that needs syncing.

Well, reducing downtime would be the next task. :) First i'd like to get it working at all.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

#5 lore raw
Quoted 10 lines · Pavel Fedin

Hello!

Quoted 5 lines

b) Once you're in the device state saving (a above) you must not change guest RAM, because at that point the migration code won't send any new changes across to the destination. So any sync's you're going to do have to happen before/at the time we stop the CPU and do the final RAM sync. On the plus side, when you're loading the device state in (a) you can be sure the RAM contents are there.

This is good. I think, in this case i can teach the kernel (here we talk about accelerated
in-kernel irqchip implementation) to flush ITS caches when a CPU is stopped. This will do the job.

Our idea at the discussion at Connect was to have an ioctl to request a flush, rather than to do it automatically when a CPU is stopped (you probably don't want to flush when only one CPU in an SMP system is stopped, for instance). But we wanted to get the basic no-ITS ABI sorted out and agreed first, so those details aren't in the patch Christoffer sent out the other day.

It will probably be more efficient if we agree on the ABI details here before doing the implementation rather than afterwards.

thanks
-- PMM

#6 lore raw

Hello!

Quoted 4 lines

Our idea at the discussion at Connect was to have an ioctl to request a flush, rather than to do it automatically when a CPU is stopped (you probably don't want to flush when only one CPU in an SMP system is stopped, for instance).

Yes, you're right. Looks like this would be more complicated but better way to do the thing. I'll look at migration state notifier mechanism.

Quoted 1 line

But we wanted to get the basic no-ITS ABI sorted out and agreed first

Yes, this is good idea, too much changes otherwise. But here at Samsung we have a project, and it has deadlines, so please understand that i try to do things quickly.
Of course, you see that as soon as i make something, i post it at least as RFCs, because i don't want to have something misdesigned, which will be dropped afterwards. Cooperation with OSS is one of goals of our project.

Thank you very much for your cooperation and explanations.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

#7 lore raw
In reply to Dr. David Alan Gilbert

Hello!

Quoted 5 lines

Some thoughts:
a) There is a migration state notifier list - see add_migration_state_change_notifier (spice
calls it)
- but I don't think it's called in the right places for your needs; we could add some more places that gets called.

I am now trying to add one more state, something like MIGRATION_STATUS_FINISHING. It would mean that CPUs are stopped. Can you explain me migration code a bit? Where is iteration loop, and where are CPUs stopped? I am looking at migration.c but
cannot say that i understand some good portion of it. :)

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

#8 lore raw
Quoted 11 lines · * Pavel Fedin ([email protected]) wrote:

Hello!

Quoted 5 lines

Some thoughts:
a) There is a migration state notifier list - see add_migration_state_change_notifier (spice
calls it)
- but I don't think it's called in the right places for your needs; we could add some more places that gets called.

I am now trying to add one more state, something like MIGRATION_STATUS_FINISHING. It would mean that CPUs are stopped. Can you explain me migration code a bit? Where is iteration loop, and where are CPUs stopped? I am looking at migration.c but
cannot say that i understand some good portion of it. :)

The outgoing side of migration comes into migrate_fd_connect which does all the setup and then starts 'migration_thread'. The big while loop in there does most of the work, and on each loop normally ends up calling either
qemu_savevm_state_iterate
or
migration_completion

migration_completion calls vm_stop_force_state to stop the CPU, and then qemu_savevm_state_complete to save all the remaining devices out.

Dave

Quoted 7 lines

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK