How long does bootstrap recovery take

The peers bootstrapper will determine whether or not it can satisfy a bootstrap request on a shard-by-shard basis by consulting the cluster placement and determining if there are enough peers to satisfy the bootstrap request.

For example, imagine the following M3DB placement where node A is trying to perform a peer bootstrap:. In this case, the peers bootstrapper running on node A will not be able to fullfill any requests because node B is in the Initializing state for all of its shards and cannot fulfill bootstrap requests. Note that a bootstrap consistency level of majority is the default value, but can be modified by changing the value of the key m3db.

For example, if an entire cluster with a replication factor of 3 was restarted simultaneously, all the nodes would get stuck in an infinite loop trying to peer bootstrap from each other and not achieving majority until an operator modified this value. Note that this can happen even if all the shards were in the Available state because M3DB nodes will reject all read requests for a shard until they have bootstrapped that shard which has to happen everytime the node is restarted. Note : Any bootstrappers configuration that does not include the peers bootstrapper will be unable to handle dynamic placement changes of any kind.

This is the only configuration that we recommend using in production. In the general case, the node will use only the filesystem and commitlog bootstrappers on node startup. Everytime a node is restarted it will attempt to stream in all of the the data for any blocks that it has never flushed, which is generally the currently active block and possibly the previous block as well. This mode can be useful if you want to improve performance or save disk space by operating nodes without a commitlog, or want to force a repair of any unflushed blocks.

In addition, if you lose a replication factors worth or more of hosts at the same time, the node will not be able to bootstrap unless an operator modifies the bootstrap consistency level configuration in etcd see peers bootstrap section above.

Finally, this mode adds additional network and resource pressure on other nodes in the cluster while one node is peer bootstrapping from them which can be problematic in catastrophic scenarios where all the nodes are trying to stream data from each other. Every time a node is restarted, it will attempt to stream in all of the data that it is responsible for from its peers, completely ignoring the immutable Fileset files it already has on disk.

This mode can be useful if you want to improve performance or save disk space by operating nodes without a commitlog, or want to force a repair of all data on an individual node. Every time a node is restarted it will utilize the immutable Fileset files its already written out to disk, but any data that it had received since it wrote out the last set of immutable files will be lost. Every time a node is restarted it will read all the commit log and snapshot files it has on disk, but it will ignore all the data in the immutable Fileset files that it has already written.

However in the case the configuration is non-default or the cluster has been down for a prolonged period of time these steps may be necessary. A good indicator would be log messages related to failing to bootstrap from peers due to consistency issues. M3DB may require manual intervention to recover in the event of a prolonged loss of quorum. This is because the Peers Boostrapper must read from a majority of nodes owning a shard to bootstrap.

Unlike regular space recovery, where a backup or replicated space performs a recovery, a bootstrap has much more implications and therefore it is an administrative command executed by the user on demand. The bootstrap is initiated on the local gateway sink of the space's site that needs to be bootstrapped for another site's space.

The following conditions should be held before initiating a bootstrap:. The space that is being bootstrapped should be first cleared of all data, a recommended way to do so is by redeploying that local space any entry which is not cleared will not be updated during the copy stage of the bootstrap process, it will be skipped. The space that is being bootstrapped should not be active during the bootstrap copy stage, which means it should not update the incoming data during the bootstrap stage.

It is up to the user to make sure this condition is held. At the bootstrap target site there should be a gateway sink configured with the bootstrap source gateway as one of its sources and there should be a delegator to that source. At the bootstrap source site there should be a gateway sink no need to configure the bootstrap target as a source if it is not supposed to replicate into the source site , and there should be a delegator from the source site to the target site.

After meting the condition specified in the previous, the bootstrap should be initiated using the Admin API. Following is an example of how to bootstrap New-York from London:. The bootstrap method will block until the bootstrap is completed and the result will specify whether the bootstrap completed successfully or some error occurred, such as timeout. When a gateway sink is started with requires-bootstrap state, it will not be open for incoming replication until a bootstrap was initiated, which means remote sites spaces incoming replication channels will be disconnected.

It is possible to enable incoming replication to a gateway sink in that state without initiating a bootstrap by calling the enableIncomingReplication. For example:.