Skip to contentSkip to conversation

colo: Add support for continious replication

PATCH Superseded Lukas Straub 3 participants 7 messages Aug 15, 2019 – Aug 21, 2019

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v2 cover colo: Add support for continious replication

Hello Everyone,
These Patches add support for continious replication to colo. Please review.

Regards,
Lukas Straub

v2:
- fix email formating
- fix checkpatch.pl warnings
- fix patchew error
- clearer commit messages

Lukas Straub (3):
Replication: Ignore requests after failover net/filter.c: Add Options to insert filters anywhere in the filter
list
Update Documentation

 block/replication.c  |  38 ++++++++-
 docs/COLO-FT.txt     | 185 ++++++++++++++++++++++++++++++++-----------
 include/net/filter.h |   2 +
 net/filter.c         |  71 ++++++++++++++++-
 qemu-options.hx      |  10 +--
 5 files changed, 250 insertions(+), 56 deletions(-)

--
2.20.1

#1 lore raw
Quoted 3 lines · * Lukas Straub ([email protected]) wrote:

Hello Everyone,
These Patches add support for continious replication to colo. Please review.

OK, for those who haven't followed COLO for so long; 'continuous replication' is when after the first primary fails, you can promote the original secondary to a new primary and start replicating again;

i.e. current COLO gives you

p<->s
<primary fails>
s

with your patches you can do

s becomes p2
p2<->s2

and you're back to being resilient again.

Which is great; because that was always an important missing piece.

Do you have some test scripts/setup for this - it would be great to automate some testing.

Dave

Quoted 22 lines

Regards,
Lukas Straub

v2:
- fix email formating
- fix checkpatch.pl warnings
- fix patchew error
- clearer commit messages

Lukas Straub (3):
Replication: Ignore requests after failover net/filter.c: Add Options to insert filters anywhere in the filter
list
Update Documentation

 block/replication.c  |  38 ++++++++-
 docs/COLO-FT.txt     | 185 ++++++++++++++++++++++++++++++++-----------
 include/net/filter.h |   2 +
 net/filter.c         |  71 ++++++++++++++++-
 qemu-options.hx      |  10 +--
 5 files changed, 250 insertions(+), 56 deletions(-)
Quoted 3 lines

--
2.20.1

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK

#2 lore raw
Quoted 21 lines · On Thu, 15 Aug 2019 19:57:37 +0100 "Dr. David Alan Gilbert"…
Quoted 3 lines · * Lukas Straub ([email protected]) wrote:

Hello Everyone,
These Patches add support for continious replication to colo. Please review.

OK, for those who haven't followed COLO for so long; 'continuous replication' is when after the first primary fails, you can promote the original secondary to a new primary and start replicating again;

i.e. current COLO gives you

p<->s
<primary fails>
s

with your patches you can do

s becomes p2
p2<->s2

Quoted 6 lines

and you're back to being resilient again.

Which is great; because that was always an important missing piece.

Do you have some test scripts/setup for this - it would be great to automate some testing.

My Plan is to write a Pacemaker Resource Agent[1] for qemu-colo and then do some long-term testing in my small cluster here. Writing standalone tests using that Resource Agent should be easy, it just needs to be provided with the right arguments and environment Variables.

Regards,
Lukas Straub

[1] https://github.com/ClusterLabs/resource-agents/blob/master/doc/dev-guides/ra-dev-guide.asc#what-is-a-resource-agent

#3 lore raw
Quoted 41 lines

-----Original Message-----
From: Lukas Straub [mailto:[email protected]]
Sent: Friday, August 16, 2019 3:48 AM
To: Dr. David Alan Gilbert <[email protected]>
Cc: qemu-devel <[email protected]>; Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Xie Changlong <[email protected]>; Wen Congyang <[email protected]>
Subject: Re: [Qemu-devel] [PATCH v2 0/3] colo: Add support for continious replication

Quoted 25 lines · On Thu, 15 Aug 2019 19:57:37 +0100 "Dr. David Alan Gilbert"…
Quoted 3 lines · * Lukas Straub ([email protected]) wrote:

Hello Everyone,
These Patches add support for continious replication to colo. Please review.

OK, for those who haven't followed COLO for so long; 'continuous replication' is when after the first primary fails, you can promote the original secondary to a new primary and start replicating again;

i.e. current COLO gives you

p<->s
<primary fails>
s

with your patches you can do

s becomes p2
p2<->s2

and you're back to being resilient again.

Which is great; because that was always an important missing piece.

Quoted 7 lines
Quoted 2 lines

Do you have some test scripts/setup for this - it would be great to automate some testing.

My Plan is to write a Pacemaker Resource Agent[1] for qemu-colo and then do some long-term testing in my small cluster here. Writing standalone tests using that Resource Agent should be easy, it just needs to be provided with the right arguments and environment Variables.

Thanks Dave's explanation.
It looks good for me and I will test this series in my side.

Another question: Is "Pacemaker Resource Agent[1] " like a heartbeat module? I have wrote an internal heartbeat module running on Qemu, it make COLO can detect fail and trigger failover automatically, no need external APP to call the QMP command "x-colo-lost-heartbeat". If you need it, I can send a RFC version recently.

Thanks
Zhang Chen

Quoted 6 lines

Regards,
Lukas Straub

[1] https://github.com/ClusterLabs/resource-agents/blob/master/doc/dev-
guides/ra-dev-guide.asc#what-is-a-resource-agent

#4 lore raw
Quoted 47 lines · Chen
Quoted 39 lines

-----Original Message-----
From: Lukas Straub [mailto:[email protected]]
Sent: Friday, August 16, 2019 3:48 AM
To: Dr. David Alan Gilbert <[email protected]>
Cc: qemu-devel <[email protected]>; Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Xie Changlong <[email protected]>; Wen Congyang <[email protected]>
Subject: Re: [Qemu-devel] [PATCH v2 0/3] colo: Add support for continious replication

Quoted 25 lines · On Thu, 15 Aug 2019 19:57:37 +0100 "Dr. David Alan Gilbert"…
Quoted 3 lines · * Lukas Straub ([email protected]) wrote:

Hello Everyone,
These Patches add support for continious replication to colo. Please review.

OK, for those who haven't followed COLO for so long; 'continuous replication' is when after the first primary fails, you can promote the original secondary to a new primary and start replicating again;

i.e. current COLO gives you

p<->s
<primary fails>
s

with your patches you can do

s becomes p2
p2<->s2

and you're back to being resilient again.

Which is great; because that was always an important missing piece.

Quoted 7 lines
Quoted 2 lines

Do you have some test scripts/setup for this - it would be great to automate some testing.

My Plan is to write a Pacemaker Resource Agent[1] for qemu-colo and then do some long-term testing in my small cluster here. Writing standalone tests using that Resource Agent should be easy, it just needs to be provided with the right arguments and environment Variables.

Quoted 4 lines

Thanks Dave's explanation.
It looks good for me and I will test this series in my side.

Another question: Is "Pacemaker Resource Agent[1] " like a heartbeat module?

It's a bit more than that. Pacemaker itself is an Cluster Resource Manager, you can think of it like sysvinit but for clusters. It controls where in the cluster Resources run, what state (master/slave) and what to do in case of a Node or Resource failure. Now Resources can be anything like SQL-Server, Webserver, VM, etc. and Pacemaker itself doesn't directly control them, that's the Job of the Resource Agents. So a Resource Agent is like an init-script, but cluster-aware with more actions like start, stop, monitor, promote (to master) or migrate-to.

Quoted 1 line

I have wrote an internal heartbeat module running on Qemu, it make COLO can detect fail and trigger failover automatically, no need external APP to call the QMP command "x-colo-lost-heartbeat". If you need it, I can send a RFC version recently.

Cool, this should be faster to failover than with Pacemaker.
What is the plan with cases like Primary-failover, which need to issue multiple commands?

Quoted 7 lines

Thanks
Zhang Chen

Quoted 5 lines

Regards,
Lukas Straub

[1] https://github.com/ClusterLabs/resource-agents/blob/master/doc/dev-guides/ra-dev-guide.asc#what-is-a-resource-agent

#5 lore raw
Quoted 78 lines

-----Original Message-----
From: Lukas Straub [mailto:[email protected]]
Sent: Saturday, August 17, 2019 2:20 AM
To: Zhang, Chen <[email protected]>
Cc: Dr. David Alan Gilbert <[email protected]>; qemu-devel <qemu- [email protected]>; Jason Wang <[email protected]>; Xie Changlong <[email protected]>; Wen Congyang <[email protected]>
Subject: Re: [Qemu-devel] [PATCH v2 0/3] colo: Add support for continious replication

Quoted 48 lines · Chen
Quoted 40 lines

-----Original Message-----
From: Lukas Straub [mailto:[email protected]]
Sent: Friday, August 16, 2019 3:48 AM
To: Dr. David Alan Gilbert <[email protected]>
Cc: qemu-devel <[email protected]>; Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Xie Changlong <[email protected]>; Wen Congyang <[email protected]>
Subject: Re: [Qemu-devel] [PATCH v2 0/3] colo: Add support for continious replication

Quoted 26 lines · On Thu, 15 Aug 2019 19:57:37 +0100 "Dr. David Alan Gilbert"…
Quoted 3 lines · * Lukas Straub ([email protected]) wrote:

Hello Everyone,
These Patches add support for continious replication to colo. Please review.

OK, for those who haven't followed COLO for so long; 'continuous replication' is when after the first primary fails, you can promote the original secondary to a new primary and start replicating again;

i.e. current COLO gives you

p<->s
<primary fails>
s

with your patches you can do

s becomes p2
p2<->s2

and you're back to being resilient again.

Which is great; because that was always an important missing piece.

Quoted 7 lines
Quoted 2 lines

Do you have some test scripts/setup for this - it would be great to automate some testing.

My Plan is to write a Pacemaker Resource Agent[1] for qemu-colo and then do some long-term testing in my small cluster here. Writing standalone tests using that Resource Agent should be easy, it just needs to be provided with the right arguments and environment Variables.

Quoted 4 lines

Thanks Dave's explanation.
It looks good for me and I will test this series in my side.

Another question: Is "Pacemaker Resource Agent[1] " like a heartbeat

module?

It's a bit more than that. Pacemaker itself is an Cluster Resource Manager, you can think of it like sysvinit but for clusters. It controls where in the cluster Resources run, what state (master/slave) and what to do in case of a Node or Resource failure. Now Resources can be anything like SQL-Server, Webserver, VM, etc. and Pacemaker itself doesn't directly control them, that's the Job of the Resource Agents. So a Resource Agent is like an init-script, but cluster- aware with more actions like start, stop, monitor, promote (to master) or migrate-to.

Quoted 8 lines
Quoted 1 line

I have wrote an internal heartbeat module running on Qemu, it make COLO

can detect fail and trigger failover automatically, no need external APP to call the QMP command "x-colo-lost-heartbeat". If you need it, I can send a RFC version recently.

Cool, this should be faster to failover than with Pacemaker.
What is the plan with cases like Primary-failover, which need to issue multiple commands?

Yes, currently we need input some net filter delete command after primary-failover. We need make a way to remove related net-filter and chardev automatically.
But for Pacemaker it isn't a problem, you can send related qmp command after the "x-lost-heart-beat".

Thanks
Zhang Chen

Quoted 10 lines
Quoted 10 lines

Thanks
Zhang Chen

Quoted 7 lines

Regards,
Lukas Straub

[1]
https://github.com/ClusterLabs/resource-agents/blob/master/doc/dev-g
uides/ra-dev-guide.asc#what-is-a-resource-agent

#6 lore raw
In reply to Lukas Straub
Quoted 42 lines · * Lukas Straub ([email protected]) wrote:
Quoted 39 lines · Chen
Quoted 39 lines

-----Original Message-----
From: Lukas Straub [mailto:[email protected]]
Sent: Friday, August 16, 2019 3:48 AM
To: Dr. David Alan Gilbert <[email protected]>
Cc: qemu-devel <[email protected]>; Zhang, Chen <[email protected]>; Jason Wang <[email protected]>; Xie Changlong <[email protected]>; Wen Congyang <[email protected]>
Subject: Re: [Qemu-devel] [PATCH v2 0/3] colo: Add support for continious replication

Quoted 25 lines · On Thu, 15 Aug 2019 19:57:37 +0100 "Dr. David Alan Gilbert"…
Quoted 3 lines · * Lukas Straub ([email protected]) wrote:

Hello Everyone,
These Patches add support for continious replication to colo. Please review.

OK, for those who haven't followed COLO for so long; 'continuous replication' is when after the first primary fails, you can promote the original secondary to a new primary and start replicating again;

i.e. current COLO gives you

p<->s
<primary fails>
s

with your patches you can do

s becomes p2
p2<->s2

and you're back to being resilient again.

Which is great; because that was always an important missing piece.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Do you have some test scripts/setup for this - it would be great to automate some testing.

My Plan is to write a Pacemaker Resource Agent[1] for qemu-colo and then do some long-term testing in my small cluster here. Writing standalone tests using that Resource Agent should be easy, it just needs to be provided with the right arguments and environment Variables.

Could you update tests/test-replication.c to test the extra steps?

Dave

Quoted 20 lines
Quoted 4 lines

Thanks Dave's explanation.
It looks good for me and I will test this series in my side.

Another question: Is "Pacemaker Resource Agent[1] " like a heartbeat module?

It's a bit more than that. Pacemaker itself is an Cluster Resource Manager, you can think of it like sysvinit but for clusters. It controls where in the cluster Resources run, what state (master/slave) and what to do in case of a Node or Resource failure. Now Resources can be anything like SQL-Server, Webserver, VM, etc. and Pacemaker itself doesn't directly control them, that's the Job of the Resource Agents. So a Resource Agent is like an init-script, but cluster-aware with more actions like start, stop, monitor, promote (to master) or migrate-to.

Quoted 1 line

I have wrote an internal heartbeat module running on Qemu, it make COLO can detect fail and trigger failover automatically, no need external APP to call the QMP command "x-colo-lost-heartbeat". If you need it, I can send a RFC version recently.

Cool, this should be faster to failover than with Pacemaker.
What is the plan with cases like Primary-failover, which need to issue multiple commands?

Quoted 7 lines

Thanks
Zhang Chen

Quoted 5 lines

Regards,
Lukas Straub

[1] https://github.com/ClusterLabs/resource-agents/blob/master/doc/dev-guides/ra-dev-guide.asc#what-is-a-resource-agent

--
Dr. David Alan Gilbert / [email protected] / Manchester, UK