// Licensed to the Apache Software Foundation (ASF) under one or more
// contributor license agreements.  See the NOTICE file distributed with
// this work for additional information regarding copyright ownership.
// The ASF licenses this file to You under the Apache License, Version 2.0
// (the "License"); you may not use this file except in compliance with
// the License.  You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
= Disaster Recovery for System Groups

A Ignite 3 cluster includes two system RAFT groups, both of which are essential for the cluster's normal operation:

* link:administrators-guide/lifecycle#cluster-management-group[Cluster Management Group (CMG)] 
* link:administrators-guide/lifecycle#cluster-metastorage-group[Metastorage Group (MG)]

You perform _disaster recovery_ operations on _system RAFT groups_ to recover permanent _majority loss_. When a system RAFT group loses majority, it becomes unavailable. When CMG is unavailable, the cluster itself remains available with limitations: it can still process most of the operations, but it cannot join new nodes, start/restart existing nodes, and start building new indexes. When MG is unavailable, the cluster becomes unusable; it cannot handle even GET/PUT/SQL requests.

NOTE: Disaster recovery for link:administrators-guide/disaster-recovery[data partitions] is described in a separate page. 

//NOTE: Other types of problems, unrelated to a majority loss, may arise - mention poisoned commands in Phase 2.

You see that the majority has been lost in cluster logs in the console or in the link:https://en.wikipedia.org/wiki/Log_rotation[rotated log files]. When a RAFT group becomes unavailable, the logs would show something like
`Send with retry timed out [retryCount = 11, groupId = cmg_group].`
or
`Send with retry timed out [retryCount = 11, groupId = metastorage_group].`.

An indicator that CMG is down is when a node does not start after a `restart` command. This is reflected in the log as `Local CMG state recovered, starting the CMG`, not followed by `Successfully joined the cluster`.

If a node tries to start when CMG is available, but MG is not, the log shows `Metastorage info on start` not followed by `Performing MetaStorage recovery`.

WARNING: Exercise caution when applying disaster recovery commands to system node groups. These commands might result in _split brain_ (split your cluster into two clusters). Use the recovery commands as the last resort, only if the majority for CMG/MG is lost permanently.

== Cluster Management Group

If CMG loses majority:

. Restart CMG nodes to restore the lost majority.
. If the above fails, forcefully assign a new majority using the following link:ignite-cli-tool#disaster-recovery-commands[CLI command] (manually or via REST): `recovery cluster reset --url=<node-url> --cluster-management-group=<new-cmg-nodes>`.
+ 
The command is sent to the node indicated by the `--url` parameter, which must belong to the `new-cmg-nodes` RAFT group. This node becomes the _Repair Conductor_, and it initiates the `reset` procedure. 
+
The above procedure might fail for the following reasons:
+
* Some of the nodes specified in `new-cmg-nodes` are not in the physical topology.
* The Repair Conductor does not have all the information it needs to start the procedure.
+
. If some nodes were down or were unavailable due to a network partition (and hence did not participate in the repair):
.. Start these nodes (or restore network connectivity and restart them).
.. Migrate these nodes to the repaired cluster using the following link:ignite-cli-tool#disaster-recovery-commands[CLI command] (manually or via REST): `recovery cluster migrate --old-cluster-url=<url-of-old-cluster-node> --new-cluster-url=<url-of-new-cluster-node>`.

== Metastorage Group

If MG loses majority:

. Restart MG nodes (or at least their RAFT nodes inside Ignite nodes).
. If the above fails:
.. Make sure that every node that could be started had started and joined the cluster.
.. Forcefully assign a new majority using the following link:ignite-cli-tool#disaster-recovery-commands[CLI command] (manually or via REST): `recovery cluster reset --url=<existing-node-url> [--cluster-management-group=<new-cmg-nodes>] --metastorage-replication-factor=N`.
+
`N` is the requested number of the voting RAFT nodes in the MG after repair. If you omit `--cluster-management-group`, the command takes the current CMG voting members set from the CMG leader; if CMG is not available, the command fails.
+
The command is sent to the node specified by `--url`. This node becomes the _Repair Conductor_, and it initiates the `reset` procedure.
+
If the Repair Conductor fails to repair MG, the procedure has to be repeated manually (there is no failover).
+
. If some nodes were down or were unavailable due to a network partition (and hence did not participate in the repair):
.. Start these nodes (or restore network connectivity and restart them).
.. Migrate these nodes to the repaired cluster using the following link:ignite-cli-tool#disaster-recovery-commands[CLI command] (manually or via REST): `recovery cluster migrate --old-cluster-url=<url-of-old-cluster-node> --new-cluster-url=<url-of-new-cluster-node>`.

NOTE: If one of the nodes has a revision in its metastorage that is not present on any of the nodes that had participated in the repair, this means that its metastorage has diverged from the MG leader's metastorage. Such a node will not be allowed to join the cluster. Its startup will fail with `MetastorageDivergedException (error code META-7)`. Remove the data from the above metastorage-divergent node and have it join the cluster as a blank node.



