// Licensed to the Apache Software Foundation (ASF) under one or more
// contributor license agreements.  See the NOTICE file distributed with
// this work for additional information regarding copyright ownership.
// The ASF licenses this file to You under the Apache License, Version 2.0
// (the "License"); you may not use this file except in compliance with
// the License.  You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
= Disaster Recovery for Data Partitions

You perform _disaster recovery_ operations to recover from situation when data operations on your Ignite cluster nodes become unfeasible because Ignite cannot guarantee data consistency. In such cases, you need to either return data to a consistent state or declare the current state consistent.

NOTE: Disaster recovery for system groups - link:administrators-guide/system-groups-recovery#cluster-management-group[Cluster Management Group] and link:administrators-guide/system-groups-recovery#metastorage-group[Metastorage Group] - is described in a separate page. 

== Disaster Scenarios and Recovery Instructions

=== Minority Offline

_Minority_ refers to less than half of the number of _replicas_ configured for a distribution zone (DZ). For example, of DZ1 is configured with 2 replicas and DZ2 - with 3 replicas, losing a single Ignite node is a majority loss for DZ1 and a minority loss for DZ2.

You may discover that one or more of your cluster nodes are offline in a number of ways, including the link:ignite-cli-tool#disaster-recovery-commands[ignite recovery partition-states] CLI command with the `--global` option, which would show `Read-only partition`, `Degraded partition`, or `Unavailable partition` for offline nodes. 

Once a minority offline status has been discovered:

. Command the system to bring the offline node(s) online.
+
The system attempts to bring the indicated nodes online. Possible outcomes are:
+
** Nodes return online in time (before scale-down timeout) with valid data. The system replicates the missing data (if any)using either log replication or the full state transfer procedure.
** Nodes return online in time but without data. The system replicates the data using the full state transfer procedure.
** A node does not return online before scale-down timeout. The system distributes a replica to a new node and starts the rebalance procedure.
** Node return online with inconsistent data - see steps 4 and 5.
+
NOTE: Full state transfer and rebalancing might take a long time (tens of minutes). We suggest that you recover nodes as soon as you discover they are offline.  
+
. Run the link:ignite-cli-tool#disaster-recovery-commands[ignite recovery partition-states] CLI command on the relevant zones/nodes/partitions to verify <<Partition States>>.
. If the state is `Healthy` or `Available partition`, consider the recovery completed.
. If the state is `Broken`:
.. Restart your GrigGain node or the relevant partitions using the link:ignite-cli-tool#disaster-recovery-commands[ignite recovery restart-partitions] CLI command.
.. Rerun the link:ignite-cli-tool#disaster-recovery-commands[ignite recovery partition-states] CLI command. 
. If the the partition state is `Read-only partition`, `Degraded partition`, or `Unavailable partition`, reset the relevant partitions using the link:ignite-cli-tool#disaster-recovery-commands[ignite recovery reset-partitions] CLI command.

=== Majority Offline

_Majority_ refers to half (or more) of the number of _replicas_ configured for a distribution zone (DZ). For example, of DZ1 is configured with 2 replicas and DZ2 - with 3 replicas, losing a single Ignite node is a majority loss for DZ1 and a minority loss for DZ2.

You may discover that one or more of your cluster nodes are offline in a number of ways, including the link:ignite-cli-tool#disaster-recovery-commands[ignite recovery partition-states] CLI command with the `--global` option, which would show `Read-only partition`, `Degraded partition`, or `Unavailable partition` for offline nodes.

If the node(s) that remain(s) online include the primary replica, the partition becomes `Read-only partition` (see <<Global Partition States>>); all the data is available for reading until lease expires.
If the node(s) that remain(s) online do _not_ include the primary replica, the partition becomes `Unavailable partition` (see <<Global Partition States>>).

Once a majority offline status has been discovered:

. Command the system to bring the offline nodes online.
+
The system attempts to bring the indicated nodes online. Possible outcomes are:
+
** Nodes return online in time (before scale-down timeout) with valid data. A leader is elected, the system replicates the missing data (if any)using either log replication or the full state transfer procedure, and a leaseholder is elected
** Nodes return online with inconsistent data - see steps 4 and 5.
+
NOTE: Full state transfer and rebalancing might take a long time (tens of minutes). We suggest that you recover nodes as soon as you discover they are offline.  
+
. Run the link:ignite-cli-tool#disaster-recovery-commands[ignite recovery partition-states] CLI command on the relevant zones/nodes/partitions to verify <<Partition States>>.
. If the state is `Healthy` or `Available partition`, consider the recovery completed.
. If the state is `Broken`: 
.. Restart your GrigGain nodes or the relevant partitions using the link:ignite-cli-tool#disaster-recovery-commands[ignite recovery restart-partitions] CLI command.
.. Rerun the link:ignite-cli-tool#disaster-recovery-commands[ignite recovery partition-states] CLI command. 
. If the the partition state is `Read-only partition`, `Degraded partition`, or `Unavailable partition`, reset the relevant partitions using the link:ignite-cli-tool#disaster-recovery-commands[ignite recovery reset-partitions] CLI command. 

In the Majority Offline scenario, you would typically lose part of the data. For example, if you reset partition A while partition B was in the `Available partition` state, you would lose:

* The latest data from A that has been restored using `reset-partitions`
* Some of the latest data from B, which had been inserted into it in a transaction that had also inserted data into A

=== Partition Loss

In this scenario, in addition to having <<Majority Offline>>, you lose all replicas of a partition, e.g., partition A. This causes a loss of _all_ the data from partition A once you run the link:ignite-cli-tool#disaster-recovery-commands[ignite recovery reset-partitions] CLI command, as well possibly a loss of some of the recent updates in other partitions.

Try bringing the nodes back online as described in the <<Majority Offline>> scenario.

== Partition States

This section describes the data partition states that define the partition availability and readiness for utilization.

=== Local Partition States

_Local partition state_ is a local property of a replica, storage, state machine, etc., associated with the partition.

* `Healthy` - a state machine is running with no issues.
* `Initializing` - a node is online, but the corresponding RAFT group has not completed its initialization yet.
* `Snapshot installation` - a full state transfer is taking place. Once it has finished, the partition will become `healthy` or `catching-up`. Before that, data cannot be read, and log replication is on pause.
* `Catching-up` - a node is in the process of replicating data from the leader, and its data is slightly in the past. More specifically, node has not replicated the tail of the log that corresponds to 100 log entries.
* `Broken` - the state machine experiences issues (likely as a result of an exception). Some data might be unavailable for reading, and the log cannot be replicated. This state will not be changed automatically - it requires intervention.

=== Global Partition States 

_Global partition state_ is a global property of a partition that specifies its apparent functionality from user's point of view.

* `Available partition` - a healthy partition that can process read and write requests. Implies that all peers are healthy at the moment.
* `Read-only partition` - a partition that can process read requests but not the write requests. There is no healthy majority. However, there is at least one alive (healthy/catch-up) peer that can process historical read-only queries.
* `Unavailable partition` - a partition that cannot process any requests.
* `Degraded partition` - a partition that is available to the user, but is at a higher risk of having issues than other partitions. For example, one of the group's peers is offline. There is still a majority, but the backup factor is low.