Periodic checkpointing (using perf and signals?)

Hi,

I'm interested in taking checkpoints of processes from fast systems like hardware and restoring them on really slow software models for performance analysis. So far I've been able to save and restore checkpoints on the different systems using CRIU. Now I'm looking for some way to trigger the checkpointing. One basic use case might be to take a process that runs for say 100M instructions and take a checkpoint every 10M instructions to be restored as 10 parallel runs of the model.

I'm thinking of trying to use performance counters to trigger such behavior. Does perf already have support for triggering things like this? If not, I'm thinking of trying to work in the ability to send a signal, like stop, to the process of interest once the specified count, such as 10M instructions, has been reached. CRIU or a wrapper could then wait for process of interest to stop, take the checkpoint, let the process continue, and then wait for it to stop again or exit. Would such an approach make sense?

Thanks,
Christopher

Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

Great idea! I will add it on http://criu.org/Usage_scenarios :)

I'm not 100% sure, but I've seen examples of python plugins for perf. From these examples, I believe that it's possible to write a plugin, that will run some code after noticing 100M instructions.

It makes perfect sense! Several things to note from my side.

1. It's perfect case where the --track-mem + --prev-images-dir options should be used. It will help subsequent dumps take MUCH less time, since with them CRIU will not take full task dump, but instead will only grab what has changed since last dump.

2. Current version of CRIU doesn't work with stopped tasks. We're currently developing it and this functionality will be available with v0.7 only. However, I think it's OK just to start "criu dump" command after perf trigger. The dump would work on a process that has done slightly more than 10M instructions, but that would be the same in case you send it STOP signal.

Thanks,
Pavel